Celery4.2在Python3.7下无法运行的问题

发表于 | 分类于 ， | 评论数： | 阅读次数：

背景

之前使用Flask + Celery + Redis来实现异步队列处理，使用的环境是python3.6，后来由于Mac系统下，使用brew安装的python，直接升级到了python3.7，相同的程序运行就报错了。

问题

报错情况提示如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
[2019-01-08 23:15:05,188: CRITICAL/MainProcess] Unrecoverable error: SyntaxError('invalid syntax', ('/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/celery/backends/redis.py', 22, 19, 'from . import async, base\n'))
Traceback (most recent call last):
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'backend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/celery/bootsteps.py", line 115, in start
    self.on_start()
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/celery/apps/worker.py", line 139, in on_start
    self.emit_banner()
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/celery/apps/worker.py", line 154, in emit_banner
    ' \n', self.startup_info(artlines=not use_image))),
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/celery/apps/worker.py", line 217, in startup_info
    results=self.app.backend.as_uri(),
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/celery/app/base.py", line 1196, in backend
    return self._get_backend()
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/celery/app/base.py", line 914, in _get_backend
    self.loader)
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/celery/app/backends.py", line 70, in by_url
    return by_name(backend, loader), url
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/celery/app/backends.py", line 50, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 860, in get_code
  File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/zhengk/.local/share/virtualenvs/ToRandom-3HChPPPT/lib/python3.7/site-packages/celery/backends/redis.py", line 22
    from . import async, base
                      ^
SyntaxError: invalid syntax

结论

这个错误有点奇怪，经过一番百度谷歌，终于发现问题的原因。由于celery中有一个文件名命名为async，而在python3.7中，新增两个关键字，其中一个恰好就是async，另一个是await。

https://docs.python.org/3/whatsnew/3.7.html

celery的作者也在issue中表示，后续版本会将async这个文件改为asynchronous，但目前版本还未发布（可能会在4.2.2）。现在可以通过github直接安装新版，pipenv环境下安装如下：

1
2
3
4
5
6
7
pipenv install https://github.com/celery/celery/tarball/master
Installing https://github.com/celery/celery/tarball/master…
✔ Installation Succeeded 
Pipfile.lock (d4f0f1) out of date, updating to (dccb00)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success!

https://github.com/celery/celery/issues/4849

拓展

flask + celery + redis 的单文件和工厂模式demo可参考如下git：

https://github.com/keejo125/flask_celery_redis_demo