Missing Python headers (Python.h) on Heroku

I’m trying to deploy to Heroku a Python application which requires some C extensions. The problem is that when I deploy the app to Heroku via git push heroku master I get the following error:

Counting objects: 9, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (9/9), 1.04 KiB | 534.00 KiB/s, done.
Total 9 (delta 3), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote: 
remote: -----> Building on the Heroku-20 stack
remote: -----> Python app detected
remote:  !     Python has released a security update! Please consider upgrading to python-3.7.10
remote:        Learn More: https://devcenter.heroku.com/articles/python-runtimes
remote: -----> Requirements file has been changed, clearing cached dependencies
remote: -----> Installing python-3.7.9
remote: -----> Installing pip 20.1.1, setuptools 47.1.1 and wheel 0.34.2
remote: -----> Installing SQLite3
remote: -----> Installing requirements with pip
remote:        Obtaining file:///tmp/build_003b7c25 (from -r /tmp/build_003b7c25/requirements.txt (line 9))
remote:        Collecting click==7.1.2
remote:          Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
remote:        Collecting Flask==1.1.2
remote:          Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
remote:        Collecting gunicorn==20.0.4
remote:          Downloading gunicorn-20.0.4-py2.py3-none-any.whl (77 kB)
remote:        Collecting itsdangerous==1.1.0
remote:          Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
remote:        Collecting Jinja2==2.11.2
remote:          Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
remote:        Collecting MarkupSafe==1.1.1
remote:          Downloading MarkupSafe-1.1.1-cp37-cp37m-manylinux2010_x86_64.whl (33 kB)
remote:        Collecting Werkzeug==1.0.1
remote:          Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
remote:        Collecting youtube-dl==2020.12.14
remote:          Downloading youtube_dl-2020.12.14-py2.py3-none-any.whl (1.8 MB)
remote:        Installing collected packages: click, Werkzeug, MarkupSafe, Jinja2, itsdangerous, Flask, gunicorn, youtube-dl, c-rename
remote:          Running setup.py develop for c-rename
remote:            ERROR: Command errored out with exit status 1:
remote:             command: /app/.heroku/python/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/build_003b7c25/setup.py'"'"'; __file__='"'"'/tmp/build_003b7c25/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
remote:                 cwd: /tmp/build_003b7c25/
remote:            Complete output (20 lines):
remote:            running develop
remote:            running egg_info
remote:            creating c_rename.egg-info
remote:            writing c_rename.egg-info/PKG-INFO
remote:            writing dependency_links to c_rename.egg-info/dependency_links.txt
remote:            writing top-level names to c_rename.egg-info/top_level.txt
remote:            writing manifest file 'c_rename.egg-info/SOURCES.txt'
remote:            reading manifest file 'c_rename.egg-info/SOURCES.txt'
remote:            writing manifest file 'c_rename.egg-info/SOURCES.txt'
remote:            running build_ext
remote:            building 'c_rename' extension
remote:            creating build
remote:            creating build/temp.linux-x86_64-3.7
remote:            creating build/temp.linux-x86_64-3.7/c_extensions
remote:            gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/app/.heroku/python/include/python3.7m -c c_extensions/rename.c -o build/temp.linux-x86_64-3.7/c_extensions/rename.o
remote:            c_extensions/rename.c:2:10: fatal error: python3.6m/Python.h: No such file or directory
remote:                2 | #include <python3.6m/Python.h>
remote:                  |          ^~~~~~~~~~~~~~~~~~~~~
remote:            compilation terminated.
remote:            error: command 'gcc' failed with exit status 1
remote:            ----------------------------------------
remote:        ERROR: Command errored out with exit status 1: /app/.heroku/python/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/build_003b7c25/setup.py'"'"'; __file__='"'"'/tmp/build_003b7c25/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
remote:  !     Push rejected, failed to compile Python app.
remote: 
remote:  !     Push failed
remote: Verifying deploy...
remote: 
remote: !   Push rejected to free-music-web.
remote: 
To https://git.heroku.com/free-music-web.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://git.heroku.com/free-music-web.git'

From what I understand, I’m missing the Python.h header file, which is required to build my C extension. I’ve already tried apt install python-dev, python-devel, python3-dev and so on, but all resulting in the same output:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
W: Not using locking for read only lock file /var/lib/dpkg/lock-frontend
W: Not using locking for read only lock file /var/lib/dpkg/lock
E: Unable to locate package python-dev

The output of cat /etc/os-release is the following

NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Now, how do I install the python headers? Thanks in advance

Answer

#include <python3.6m/Python.h>

This looks to me like a badly-written C extension which is hardcoded to only work on “Python 3.6 with PyMalloc”. Since you’re using Python 3.7.9 the path obviously doesn’t exist.

The usual approach is to

#include <Python.h>

When setuptools/distutils builds a module it sets the include path appropriately (you can see that it’s done this with -I/app/.heroku/python/include/python3.7m in your output).

Personally I’d have my doubts about any C extension that hardcodes the paths like this – it’s such an odd thing to do that it doesn’t leave you with much faith in the quality of the rest of the code. It’s also likely that this extension has only been tested or designed to work on Python 3.6, so there’s no guarantee it works on Python 3.7 (generally there aren’t huge compatibility changes between versions, but then we already know that this extension writer has made some odd decisions…)