django restframework+fastdfs 或者Minio实现django自定义存储，上传文件

lewis 1年前 (2024-04-16) 阅读数 13 #技术

fastdfs client库windows调用报错报错坑1：

ModuleNotFoundError: No module named 'mutagen._compat'
解决办法：找到utils.py文件修复导入报问题如下：

from mutagen._senf._compat import StringIO

坑2：进行settings.py client.conf配置路径settings.FASTDFS_CLIENT_CONF引入报错,实际上我已经配置了在settings.py很狗血：

Requested setting , but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

解决办法：

import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mall.settings')

坑三：fastdfs 使用方法client.upload_by_file()报错Error: 'sendfile' system call only available on linux.此方法只能在linux系统使用windows使用错误

client=Fdfs_client(conf_path=conf)
client.upload_by_file()

解决办法：使用client.upload_by_buffer(data.read())方法调试：

conf=settings.FASTDFS_CLIENT_CONF
    client=Fdfs_client(conf_path=conf)
    with open('hi.png','rb')as data:
        ret=client.upload_by_buffer(data.read())
        print(ret)
        if ret.get("Status")!="Upload successed.":
            raise Exception('fastdfs upload failed !!!')
        file_name = ret.get("Remote file_id")
        print(file_name)

　fastdfs 自定义存储实现如下：

import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mall.settings')
from django.conf import settings
from django.core.files.storage import Storage
from django.utils.deconstruct import deconstructible
from fdfs_client.client import Fdfs_client

@deconstructible
class FastDfsStorage(Storage):
    """
    org-doc: https://www.pianshen.com/article/1299301803/
    :param server_url: fast-dfs服务器ip:port
    :param: client_conf
    """

    def __init__(self, server_url=None, client_conf=None):
        if not server_url:
            self.server_url = server_url
        self.server_url = settings.FASTDFS_URL
        if not client_conf:
            self.client_conf = client_conf
        self.client_conf = settings.FASTDFS_CLIENT_CONF

    def _open(self, name, mode='rb'):
        """ for read file : Retrieve the specified file from storage."""
        return super().open(name,'rb')

    def _save(self, name, content, max_length=None):
        client=Fdfs_client(conf_path=self.client_conf)
        extend_name=name.split('.')[-1]
        ret=client.upload_by_buffer(content.read(),file_ext_name=extend_name)
        if ret.get("Status")!="Upload successed.":
            raise Exception('fastdfs upload failed !!!')
        file_name = ret.get("Remote file_id").replace('\\','/')
        print('测试获取文件路径',file_name)
        return file_name

    def url(self, name):
        path=self.server_url+'/'+name
        return path

    def exists(self, name):
        """ means filename always is available is new filename, fast-dfs always  storage one same file """
        return False

接下来,使用drf编写模型类，序列化器以及view视图测试代码：

model如下：

class FileModels(models.Model):
    file_id=models.AutoField(primary_key=True,max_length=200)
    file=models.FileField()

    class Meta:
        db_table='tbl_file'
        verbose_name='文件测试'

序列化器如下：

from rest_framework import serializers
from .models import FileModels


class SerialFiles(serializers.ModelSerializer):
    class Meta:
        model = FileModels
        fields = "__all__"

视图：

class FileView(APIView):

    def post(self, request, *args, **kwargs):
            print(request.POST)
            print(request.data)
            ser = SerialFiles(data=request.data, many=False)
            if ser.is_valid():
                ser.save()
                return Response(data=ser.data, status=status.HTTP_200_OK)
            else:
                return Response(data=ser.errors, status=status.HTTP_404_NOT_FOUND)

    def get(self,request, *args, **kwargs):
        inst=FileModels.objects.all()
        ser=SerialFiles(instance=inst,many=True)

        return Response(data=ser.data)

注意事项起初，我用postman form-data 上传一直报参数f不存在，匪夷所思，怀疑是没有指明使用muti-formdata导致的，于是使用html写个简单的上传表单提交文件，果然

最后点击上传可以看到页面返回：

最后送上我的表单html上传文件基于 enctype=

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
</head>
<body>
    <h1><div align=center style="color:green" display="block"> hello world </div><p> this is a first text for test web page and we will sart from hello </p></h1>
<form action="http://127.0.0.1:8000/file/" method="post" enctype="multipart/form-data">

    <input type="file"  name='file'>
    <!--input type="file"  name='file22'-->
    <!--input type="username"  name='username'-->
    <!--input type="password"  name='password'-->
     <!--input type="file"  accept=".xlsx"-->
    <input type="submit" value="提交">

</form>
</body>
© codemao
</html>

结语：其实自定义文件储存，在企业使用是非常多的oss,fastdfs,minio ,qiniu云，腾讯cos等等，django默认自带FilesystemStorage，实现一般存在django项目所在服务器时间久了影响磁盘空间以及性能，企业很少这样做实际开发

关于minio自定义存储可以这样：

可能遇到的坑：

第一：settings 定义MINIO_CONF找不到，我用的django2.2，好像必须要大写变量在settings

事实上我已经导入了

import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mall.settings')

坑2：save方法返回name是文件名，不然找不到，其次我在上传minio是object_name修改了用了uuid加随机字符串避免多用户使用上传文件名相同覆盖问题

def _save(self, name, content, max_length=None):

        if not self.minio_client.bucket_exists(self.bucket_name):
            self.minio_client.make_bucket(self.bucket_name)
        try:
            self.minio_client.get_bucket_policy(self.bucket_name)
        except NoSuchBucketPolicy:
            self.set_bucket_policy_public(self.bucket_name)
        (etag, vid)=self.minio_client.put_object(bucket_name=self.bucket_name,object_name=name,data=content,length=content.size)
        if etag:
            print("save return  minio is {}".format('/{}/{}'.format(self.bucket_name,name)))
            # 这里返回name,就是url方法的入参name
            return  name

送上settings配置：

DEFAULT_FILE_STORAGE='utils.minio_fdfs.minio_storage.MinioStorage'
MINIO_CONF = {
    'endpoint': '192.168.110.151:9000',
    'access_key': 'admin',
    'secret_key': 'admin123456',
    'secure': False
}
BUCKET_NAME='mybucket'

import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mall.settings')
from django.conf import settings
from django.core.files.storage import Storage
from django.utils.deconstruct import deconstructible
from minio import Minio, ResponseError
from minio.error import NoSuchBucketPolicy
import json
import uuid
import random
import string

@deconstructible
class MinioStorage(Storage):
    """
    python docs:  https://docs.min.io/docs/python-client-api-reference.html
    @:param fp : fileobject
    @:param object_name is the object name which will save to  minio bucket
    @:param bucket_name the bucket name of minio .
    """

    def __init__(self, bucket_name=None, object_name=None):
        if not settings.MINIO_CONF:
            raise ValueError('required MINIO_CONF  config in django.settings: format is:\n{}'.format(
                {'endpoint': '192.168.110.151:9000',
                 'access_key': 'username',
                 'secret_key': 'password',
                 'secure': False,
                 }
            ))
        self.minio_conf = settings.MINIO_CONF
        if not settings.BUCKET_NAME:
            self.bucket_name = bucket_name
        self.bucket_name=settings.BUCKET_NAME
        self.object_name = object_name
        self.endpoint_minio=settings.MINIO_CONF.get('endpoint',None)
        self.minio_client= Minio(**self.minio_conf)
    def _open(self, name, mode='rb'):
        """ for read file : Retrieve the specified file from storage."""
        return super().open(name, 'rb')

    def _save(self, name, content, max_length=None):
        filename, extendname = name.split('.')
        salt= ''.join(random.sample(string.ascii_letters + string.digits, 7))
        name = name.replace(filename, uuid.uuid1().hex +salt+'_' +filename)
        if not self.minio_client.bucket_exists(self.bucket_name):
            self.minio_client.make_bucket(self.bucket_name)
        try:
            self.minio_client.get_bucket_policy(self.bucket_name)
        except NoSuchBucketPolicy:
            self.set_bucket_policy_public(self.bucket_name)
        (etag, vid)=self.minio_client.put_object(bucket_name=self.bucket_name,object_name=name,data=content,length=content.size)
        if etag:
            # print("save return  minio is {}".format('/{}/{}'.format(self.bucket_name,name)))
            # 这里返回name,就是url方法的入参name
            name_file='{}/{}'.format(self.bucket_name,name)
            return name_file

    def url(self, name):
        print("url minio return is {}".format(self.endpoint_minio +'/{}/{}'.format(self.bucket_name,name)))
        return self.endpoint_minio +'/'+name

    def exists(self, name):
        """ means filename always is available is new filename, fast-dfs always  storage one same file """
        return False

    def set_bucket_policy_public(self,bucket_name):
        """set file to public download by url: http:endpoint/bucket_name/object-name"""

        policy = {'Version': '2012-10-17', 'Statement': [{'Effect': 'Allow',
                                                          'Principal': {'AWS': ['*']},
                                                          'Action': ['s3:GetBucketLocation', 's3:ListBucket'],
                                                          'Resource': ['arn:aws:s3:::{}'.format(bucket_name)]},
                                                         {'Effect': 'Allow', 'Principal': {'AWS': ['*']},
                                                          'Action': ['s3:GetObject'],
                                                          'Resource': ['arn:aws:s3:::{}/*'.format(bucket_name)]}]}
        # set  bucket to public download
        self.minio_client.set_bucket_policy(bucket_name=bucket_name, policy=json.dumps(policy))

    def get_minio_object(self,bucket_name,object_name):
        """ :return minio object of file,if get bytes by read()"""
        response=None
        try:
            response=self.minio_client.get_object(bucket_name=bucket_name,object_name=object_name)
        finally:
            response.close()
            response.release_conn()

去minio查看上传情况：

下一篇docker 部署minio，对比了下fastdfs，我还是喜欢minio，真的很爽：

数据库存储可以带文件拓展名；第一条是minio，第二条fastdfs

版权声明

本文仅代表作者观点，不代表博信信息网立场。

上一篇：ROPE怎么处理对抗性样本下一篇：javascript - BS结构下使用JS开发NFC读写器

django restframework+fastdfs 或者Minio实现django自定义存储，上传文件

版权声明

作者文章