cimport cython module that wraps C libraries

Question

I have this kind of C/cython project:

project/
├── src/
│   └── modules/
│       ├── cython1.pyx
│       ├── cython1.pxd
│       ├── cython2.pyx
│       ├── cython2.pxd
│       ├── includes/
│       │   ├── c1.h
│       │   ├── c1.c
│       │   ├── c2.h
│       │   ├── c2.c
│       └── ...
└── setup.py

with this setup.py:

filenames = ['cython1', 'cython2']
extensions = [Extension(name=f"modules.{name}", 
                        sources=[f"modules/{name}.pyx"],
                        include_dirs=["modules/includes", ]) for name in filenames]

setup(
    ext_modules=cythonize(extensions),
    packages=['project'],
    include_package_data=True,
    package_data = {
        'module': ['*.pxd', '*.so'],
    },
    ]
)

I have several issues.

In pxd files, I use definitions from C files as follows: cdef extern from "includes/c1.c" and the project compiles and runs without any issue. When I try to cimport module1 in another context, I have a fatal error: 'c1.c' file not found.

When I setup cdef extern from "includes/c1.h" in the pxd files, the project compiles but its execution gives errors (symbol not found in flat namespace : functions from C files are not in namespace).

I tried to add the corresponding C files to the source list for each module (sources=[f"modules/{name}.pyx", "c1.c"]), then I get a message saying that C functions are declared multiple times. The fact is that I use C functions in the different cython modules. Moreover, the different c/h files interact with each other (for instance there can be an #include "c1.h" in "c2.c").

In the end, I just can't manage it, and can't figure out how to structure the project so that it runs properly and can also be cimported. I understand from the forums that one solution might be to precompile my C file as a shared library, but I can't get this procedure to work directly in setup.py.

What's the best way to structure the project and to write the setup.py?

EDIT :

I change the structure of the project:

project/
├── src/
│   ├── modules_clib/
│   │   ├── c1.h
│   │   ├── c1.c
│   │   ├── c2.h
│   │   └── c2.c
│   │
│   └── modules/
│       ├── cython1.pyx
│       ├── cython1.pxd
│       ├── cython2.pyx
│       ├── cython2.pxd
│       └── includes.pxd
└── setup.py

and the setup.py to this:

filenames = ['cython1', 'cython2']
extensions = [Extension(name=f"modules.{name}", 
                        sources=[f"./src/modules/{name}.pyx"],
                        include_dirs=["./src/modules_clib", ]) for name in filenames]

setup(
    ext_modules=cythonize(extensions),
    packages=['project'],
    include_package_data=True,
    packages=['modules_clib', 'modules'],
    package_dir={'': 'src'},
    package_data={'modules_clib': ['*.c', '*.h'],
                  'modules': ['*.pxd', '*.so']},
    ]
)

All the cdef extern from "../modules_clib/c1.c" are now gathered in the includes.pxd file. In the moduleX.pyx files, I cimport all necessary C functions from modules.includes. I can now also cimport modules.includes or cimport modules.cythonX from another project/notebook to use C functions since libc is copied in the site-packages directory. I don't know if it is a good thing that libc is copied directly in the site-packagesdirectory, but it works.

However, I'm still forced to use C files directly. Using headers still leads to symbol not found in flat namespace. From what I understand, when C files are used, cython compiles them directly with the *.pyx files that use them, which is not the case if headers are used.