APIcast icon indicating copy to clipboard operation
APIcast copied to clipboard
verified publisher icon 3scale

Setup local env fails

Open imod opened this issue 3 years ago • 10 comments

Hi,

I'm trying to setup a local environment to develop a custom policy, but I fail to get APIcast up and running as described...

Version

master branch

Steps To Reproduce
  1. git clone [email protected]:3scale/APIcast.git
  2. cd APIcast
  3. make development
  4. make dependencies --> fails
Current Result
domi 🙈 APIcast $ make development
Running on Darwin
docker compose -f docker-compose-devel.yml -f docker-compose-devel-volmount-mac.yml up -d
[+] Running 2/2
 ⠿ Container apicast_build_0-redis-1        Running                                                                            0.0s
 ⠿ Container apicast_build_0-development-1  Started                                                                            1.5s
docker compose -f docker-compose-devel.yml -f docker-compose-devel-volmount-mac.yml exec -e COLUMNS="`tput cols`" -e LINES="`tput lines`" --user 501:20 development bash
bash-4.4$ make dependencies
/usr/local/openresty/luajit/bin/rover lock --roverfile=/opt/app-root/src/gateway/Roverfile
Resolving /opt/app-root/src/gateway/Roverfile
Cloning into 'lua-resty-repl'...
fatal: unable to connect to github.com:
github.com[0: 140.82.121.3]: errno=Connection refused

/usr/local/openresty/luajit/bin/luajit-2.1.0-beta3: /usr/share/lua/5.1/rover/lock.lua:169: attempt to index local 'rockspec' (a nil value)
stack traceback:
        /usr/share/lua/5.1/rover/lock.lua:169: in function 'expand_dependencies'
        /usr/share/lua/5.1/rover/lock.lua:199: in function 'resolve'
        /usr/share/lua/5.1/rover/cli/lock.lua:23: in function </usr/share/lua/5.1/rover/cli/lock.lua:11>
        /usr/share/lua/5.1/rover/cli.lua:33: in function </usr/share/lua/5.1/rover/cli.lua:27>
        /usr/lib64/luarocks/rocks/lua-rover/scm-1/bin/rover:8: in main chunk
        [C]: at 0x55ad138087a0
make: *** [Makefile:214: /opt/app-root/src/gateway/Roverfile.lock] Error 1
bash-4.4$ ^C
bash-4.4$ exit
Additional Information
  • I tried to access github.com from within the running container, it works with git (ssh and https) and curl (https)

imod avatar Sep 30 '22 11:09 imod

That is because some lua deps have git://github.com/ as source and github, for security reasons removed git plain protocol.

The proper fix should be done in the lua deps repos, replacing git://github.com/ with [email protected] or ssh://github.com. In the meantime, the workaround is to update your git configuration, i.e. in .gitconfig adding

[url "https://github.com/"]
	insteadOf = git://github.com/

That should do it

eguzki avatar Sep 30 '22 13:09 eguzki

thanks @eguzki ! It does get me further, but now it fails with this...

domi 🙈 APIcast $ make development
Running on Darwin
docker-compose -f docker-compose-devel.yml -f docker-compose-devel-volmount-mac.yml up -d
Creating network "apicast_build_0_default" with the default driver
Creating apicast_build_0_redis_1 ... done
Creating apicast_build_0_development_1 ... done
docker-compose -f docker-compose-devel.yml -f docker-compose-devel-volmount-mac.yml exec -e COLUMNS="`tput cols`" -e LINES="`tput lines`" --user 501:20 development bash
bash-4.4$ git config --global url.https://github.com/.insteadOf git://github.com/
bash-4.4$ cat ~/.gitconfig 
[url "https://github.com/"]
	insteadOf = git://github.com/
bash-4.4$ make dependencies
/usr/local/openresty/luajit/bin/rover lock --roverfile=/opt/app-root/src/gateway/Roverfile
Resolving /opt/app-root/src/gateway/Roverfile
Cloning into 'lua-resty-repl'...
remote: Enumerating objects: 52, done.
remote: Counting objects: 100% (52/52), done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 52 (delta 3), reused 36 (delta 0), pack-reused 0
Unpacking objects: 100% (52/52), 21.30 KiB | 474.00 KiB/s, done.
Note: switching to '3878f41b7e8f97b1c96919db19dbee9496569dda'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

/usr/local/openresty/luajit/bin/luajit-2.1.0-beta3: /usr/share/lua/5.1/rover/lock.lua:169: attempt to index local 'rockspec' (a nil value)
stack traceback:
	/usr/share/lua/5.1/rover/lock.lua:169: in function 'expand_dependencies'
	/usr/share/lua/5.1/rover/lock.lua:199: in function 'resolve'
	/usr/share/lua/5.1/rover/cli/lock.lua:23: in function </usr/share/lua/5.1/rover/cli/lock.lua:11>
	/usr/share/lua/5.1/rover/cli.lua:33: in function </usr/share/lua/5.1/rover/cli.lua:27>
	/usr/lib64/luarocks/rocks/lua-rover/scm-1/bin/rover:8: in main chunk
	[C]: at 0x55ceb66087a0
make: *** [Makefile:214: /opt/app-root/src/gateway/Roverfile.lock] Error 1
bash-4.4$

imod avatar Oct 03 '22 04:10 imod

Actually, my hint about the git configuration is not needed, because the Makefile already does that for us (target translate_git_protocol called as dependecy of the target make dependencies. So I do not know why helped a bit.

The error is about trying to load a dependecy that is not found

Which commitID of the master branch are you working on?

Try to run make clean that will remove local docker images so you make sure images being used are latest.

eguzki avatar Oct 03 '22 08:10 eguzki

Ok, this is something related to macos. We will fix that.

eguzki avatar Oct 03 '22 08:10 eguzki

@eguzki great, many thanks! can you please ping me once this is available?

imod avatar Oct 03 '22 10:10 imod

unfortunate I don't understand the issue (yet), but is there anything I can help with?

imod avatar Oct 04 '22 06:10 imod

I am waiting to have one macOS so I can dig into the issue. Some lua library is not being installed and the lua manager, rover, does not check when a dep is not found and when accessing the name, it crashes. That's all I can say until I can reproduce in a macOS and figure out which dep is not being found and why

eguzki avatar Oct 04 '22 07:10 eguzki

@eguzki maybe I can help you with some more details? just let me know what information you require and I'm happy to reproduce and provide it. I really would like to get started with custom policies :)

imod avatar Oct 12 '22 05:10 imod

I did some investigation on this today. Here are a couple changes to workaround the issues reported (until a proper fix is sent).

1. "Connection refused"

To avoid Connection refused and the extra git config --global url.https://github.com/.insteadOf git://github.com command, which should not be needed.

Error:

bash-4.4$ make dependencies
/usr/local/openresty/luajit/bin/rover lock --roverfile=/opt/app-root/src/gateway/Roverfile
Resolving /opt/app-root/src/gateway/Roverfile
Cloning into 'lua-resty-repl'...
fatal: unable to connect to github.com:
github.com[0: 140.82.121.3]: errno=Connection refused

/usr/local/openresty/luajit/bin/luajit-2.1.0-beta3: /usr/share/lua/5.1/rover/lock.lua:169: attempt to index local 'rockspec' (a nil value)
stack traceback:
	/usr/share/lua/5.1/rover/lock.lua:169: in function 'expand_dependencies'
	/usr/share/lua/5.1/rover/lock.lua:199: in function 'resolve'
	/usr/share/lua/5.1/rover/cli/lock.lua:23: in function </usr/share/lua/5.1/rover/cli/lock.lua:11>
	/usr/share/lua/5.1/rover/cli.lua:33: in function </usr/share/lua/5.1/rover/cli.lua:27>
	/usr/lib64/luarocks/rocks/lua-rover/scm-1/bin/rover:8: in main chunk
	[C]: at 0x40000087a0
make: *** [Makefile:214: /opt/app-root/src/gateway/Roverfile.lock] Error 1

Fix:

diff --git a/Makefile b/Makefile
index dc8a667d..d1586798 100644
--- a/Makefile
+++ b/Makefile
@@ -216,7 +216,7 @@ $(GATEWAY_CONTEXT)/Roverfile.lock : $(GATEWAY_CONTEXT)/Roverfile $(GATEWAY_CONTE
 translate_git_protocol:
 	@git config --global url.https://github.com/.insteadOf git://github.com/
 
-lua_modules: $(ROVER) $(GATEWAY_CONTEXT)/Roverfile.lock translate_git_protocol
+lua_modules: $(ROVER) translate_git_protocol $(GATEWAY_CONTEXT)/Roverfile.lock
 # This variable is to skip issues with openssl 1.1.1
 # https://github.com/wahern/luaossl/issues/175
 	EXTRA_CFLAGS="-DHAVE_EVP_KDF_CTX=1" $(ROVER) install --roverfile=$(GATEWAY_CONTEXT)/Roverfile > /dev/null

2. "Rockspec format 3.0 not supported" (lua-resty-jwt v0.2.3) – the root cause of the last issue reported by @imod

To avoid attempt to index local 'rockspec' (a nil value), after fixing the git scheme and running make dependencies; caused by lua-resty-jwt with of the addition of lua-resty-openssl in v0.2.3 (see https://github.com/cdbattags/lua-resty-jwt/pull/44).

Error:

bash-4.4$ make dependencies
/usr/local/openresty/luajit/bin/rover lock --roverfile=/opt/app-root/src/gateway/Roverfile
Resolving /opt/app-root/src/gateway/Roverfile
/usr/local/openresty/luajit/bin/luajit-2.1.0-beta3: /usr/share/lua/5.1/rover/lock.lua:169: attempt to index local 'rockspec' (a nil value)
stack traceback:
	/usr/share/lua/5.1/rover/lock.lua:169: in function 'expand_dependencies'
	/usr/share/lua/5.1/rover/lock.lua:199: in function 'resolve'
	/usr/share/lua/5.1/rover/cli/lock.lua:23: in function </usr/share/lua/5.1/rover/cli/lock.lua:11>
	/usr/share/lua/5.1/rover/cli.lua:33: in function </usr/share/lua/5.1/rover/cli.lua:27>
	/usr/lib64/luarocks/rocks/lua-rover/scm-1/bin/rover:8: in main chunk
	[C]: at 0x40000087a0
make: *** [Makefile:214: /opt/app-root/src/gateway/Roverfile.lock] Error 1

Fix:

diff --git a/gateway/apicast-scm-1.rockspec b/gateway/apicast-scm-1.rockspec
index bac8c66a..bc7999e0 100644
--- a/gateway/apicast-scm-1.rockspec
+++ b/gateway/apicast-scm-1.rockspec
@@ -14,7 +14,7 @@ dependencies = {
    'inspect',
    'lyaml',
    'router',
-   'lua-resty-jwt',
+   'lua-resty-jwt == 0.2.0',
    'lua-resty-url',
    'lua-resty-env',
    'lua-resty-execvp',

3. "Could not find header file for PCRE" (jsonschema v0.9.8)

To avoid Could not find header file for PCRE (not reported here), after rover lock passes and rover install is executed by the lua_modules target; caused by jsonschema with the addition of lrexlib-pcre in v0.9.8 (see https://github.com/api7/jsonschema/pull/76).

Error:

bash-4.4$ make dependencies
Resolved Roverfile.lock in 	142	s
EXTRA_CFLAGS="-DHAVE_EVP_KDF_CTX=1" /usr/local/openresty/luajit/bin/rover install --roverfile=/opt/app-root/src/gateway/Roverfile > /dev/null
[...]
/usr/local/openresty/luajit/bin/luajit-2.1.0-beta3: /usr/share/lua/5.1/rover/install.lua:59: Could not find header file for PCRE
  No file pcre.h in /usr/local/include
  No file pcre.h in /usr/include
You may have to install PCRE in your system and/or pass PCRE_DIR or PCRE_INCDIR to the luarocks command.
Example: luarocks install lrexlib-pcre PCRE_DIR=/usr/local
stack traceback:
	[C]: in function 'assert'
	/usr/share/lua/5.1/rover/install.lua:59: in function 'install'
	/usr/share/lua/5.1/rover/install.lua:102: in function 'call'
	/usr/share/lua/5.1/rover/cli/install.lua:42: in function </usr/share/lua/5.1/rover/cli/install.lua:19>
	/usr/share/lua/5.1/rover/cli.lua:33: in function </usr/share/lua/5.1/rover/cli.lua:27>
	/usr/lib64/luarocks/rocks/lua-rover/scm-1/bin/rover:8: in main chunk
	[C]: at 0x40000087a0
make: *** [Makefile:222: lua_modules] Error 1

Fix:

diff --git a/gateway/Roverfile b/gateway/Roverfile
index a331fe06..b0f16e93 100644
--- a/gateway/Roverfile
+++ b/gateway/Roverfile
@@ -9,7 +9,7 @@ luarocks {
     group 'testing' {
         module { 'busted' },
         module { 'luacov' },
-        module { 'jsonschema' },
+        module { 'jsonschema', '0.8' },
     },
 
     group 'development' {

I have tested the above on macOS with M1 chip and the fixes make make dependencies pass. However, neither I have checked for regression on Linux, nor I can tell just yet if this is enough for the development environment to be fully functional on macOS.

After applying all the 3 fixes above and running make dependencies inside the development container, you should see the following output:

Complete! Modules were bundled into /opt/app-root/src/vendor/cache

In time: I also observed the following error when trying to compile luaossl at one of my attempts:

bash-4.4$ make dependencies
[...]
Resolved Roverfile.lock in 	142	s
EXTRA_CFLAGS="-DHAVE_EVP_KDF_CTX=1" /usr/local/openresty/luajit/bin/rover install --roverfile=/opt/app-root/src/gateway/Roverfile > /dev/null
Warning: skipping dependency checks.
No existing manifest. Attempting to rebuild...
Warning: skipping dependency checks.
Warning: skipping dependency checks.
Warning: skipping dependency checks.
Hunk 1 found at offset 2...
successfully patched /tmp/luarocks_luarocks-source-luaossl-20220711-0-oCpRgw/luaossl-rel-20220711/src/openssl.c
src/openssl.c:623:10: fatal error: openssl/core_names.h: No such file or directory
 #include <openssl/core_names.h>
          ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
/usr/local/openresty/luajit/bin/luajit-2.1.0-beta3: /usr/share/lua/5.1/rover/install.lua:59: Build error: Failed compiling object src/openssl.o
stack traceback:
	[C]: in function 'assert'
	/usr/share/lua/5.1/rover/install.lua:59: in function 'install'
	/usr/share/lua/5.1/rover/install.lua:102: in function 'call'
	/usr/share/lua/5.1/rover/cli/install.lua:42: in function </usr/share/lua/5.1/rover/cli/install.lua:19>
	/usr/share/lua/5.1/rover/cli.lua:33: in function </usr/share/lua/5.1/rover/cli.lua:27>
	/usr/lib64/luarocks/rocks/lua-rover/scm-1/bin/rover:8: in main chunk
	[C]: at 0x40000087a0
make: *** [Makefile:222: lua_modules] Error 1

Exiting the container and running make clean, make development, and then make dependencies again solved the issue for me.

guicassolato avatar Oct 17 '22 15:10 guicassolato

I have run on Linux with your fixes (in details), but unfortunately I have the issue

src/openssl.c:623:10: fatal error: openssl/core_names.h: No such file or directory
 #include <openssl/core_names.h>
          ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

I cannot solve the issue running make clean, make development and then make dependencies.

iff --git a/Makefile b/Makefile
index dc8a667d..d1586798 100644
--- a/Makefile
+++ b/Makefile
@@ -216,7 +216,7 @@ $(GATEWAY_CONTEXT)/Roverfile.lock : $(GATEWAY_CONTEXT)/Roverfile $(GATEWAY_CONTE
 translate_git_protocol:
 	@git config --global url.https://github.com/.insteadOf git://github.com/
 
-lua_modules: $(ROVER) $(GATEWAY_CONTEXT)/Roverfile.lock translate_git_protocol
+lua_modules: $(ROVER) translate_git_protocol $(GATEWAY_CONTEXT)/Roverfile.lock
 # This variable is to skip issues with openssl 1.1.1
 # https://github.com/wahern/luaossl/issues/175
 	EXTRA_CFLAGS="-DHAVE_EVP_KDF_CTX=1" $(ROVER) install --roverfile=$(GATEWAY_CONTEXT)/Roverfile > /dev/null
diff --git a/gateway/Roverfile b/gateway/Roverfile
index a331fe06..b0f16e93 100644
--- a/gateway/Roverfile
+++ b/gateway/Roverfile
@@ -9,7 +9,7 @@ luarocks {
     group 'testing' {
         module { 'busted' },
         module { 'luacov' },
-        module { 'jsonschema' },
+        module { 'jsonschema', '0.8' },
     },
 
     group 'development' {
diff --git a/gateway/Roverfile.lock b/gateway/Roverfile.lock
index 055ca7a6..72e914f6 100644
--- a/gateway/Roverfile.lock
+++ b/gateway/Roverfile.lock
@@ -1,40 +1,40 @@
-argparse 0.6.0-1||production
-busted 2.0.rc12-1||testing
+argparse 0.7.1-1||production
 basexx 0.4.1-1||development
 binaryheap 0.4-1||development
-bit32 5.3.0-1||development
-compat53 0.7-1||development
-cqueues 20190813.51-0||development
-dkjson 2.5-2||testing
+bit32 5.3.5.1-1||development
+busted 2.1.1-1||testing
+compat53 0.8-1||development
+cqueues 20200726.51-0||development
+date 2.2-2||production
+dkjson 2.6-1||testing
 fifo 0.2-0||development
-http 0.3-0||development
-inspect 3.1.1-0||production
+http 0.4-0||development
+inspect 3.1.3-0||production
+jsonschema 0.8-0|c1d72d86bb3dc5b33da57d47febc47657d29ea74|testing
 ldoc 1.4.6-2||development
-date 2.2-2||production
 liquid 0.2.0-2||production
-jsonschema 0.8-0|aa4740624cca4c10585bd7d086b42aa0b9ab14fa|testing
 lpeg 1.0.2-1||development
 lpeg_patterns 0.5-0||development
 lua-resty-env 0.4.0-1||production
 lua-resty-execvp 0.1.1-1||production
-lua-resty-http 0.15-0||production
-lua-resty-iputils 0.3.0-1||production
+lua-resty-http 0.17.0.beta.1-0||production
+lua-resty-iputils 0.3.0-2||production
 lua-resty-jit-uuid 0.0.7-2||production
 lua-resty-jwt 0.2.0-0||production
 lua-resty-repl 0.0.6-0|3878f41b7e8f97b1c96919db19dbee9496569dda|development
 lua-resty-url 0.3.5-1||production
 lua-term 0.7-1||testing
 lua_cliargs 3.0-2||testing
-luacov 0.13.0-1||testing
-luafilesystem 1.7.0-2||production,development,testing
-luaossl 20190731-0||development
-luassert 1.8.0-0||testing
+luacov 0.15.0-1||testing
+luafilesystem 1.8.0-1||testing,production,development
+luaossl 20220711-0||development
+luassert 1.9.0-1||testing
 luasystem 0.2.1-0||testing
-lyaml 6.2.4-1||production
+lyaml 6.2.7-1||production
 markdown 0.33-1||development
 mediator_lua 1.1.2-0||testing
-net-url 0.9-1||testing
-nginx-lua-prometheus 0.20181120-3||production
-penlight 1.7.0-1||production,development,testing
+net-url 1.1-1||testing
+nginx-lua-prometheus 0.20220527-1||production
+penlight 1.13.1-1||testing,production,development
 router 2.1-0||production
-say 1.3-1||testing
+say 1.4.1-3||testing
\ No newline at end of file
diff --git a/gateway/apicast-scm-1.rockspec b/gateway/apicast-scm-1.rockspec
index bac8c66a..bc7999e0 100644
--- a/gateway/apicast-scm-1.rockspec
+++ b/gateway/apicast-scm-1.rockspec
@@ -14,7 +14,7 @@ dependencies = {
    'inspect',
    'lyaml',
    'router',
-   'lua-resty-jwt',
+   'lua-resty-jwt == 0.2.0',
    'lua-resty-url',
    'lua-resty-env',
    'lua-resty-execvp',

eguzki avatar Oct 17 '22 16:10 eguzki

I have run on Linux with your fixes (in details), but unfortunately I have the issue

src/openssl.c:623:10: fatal error: openssl/core_names.h: No such file or directory
 #include <openssl/core_names.h>
          ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

I cannot solve the issue running make clean, make development and then make dependencies.

@eguzki, you're right.

This is caused by the luaossl module version 20220711, specifically https://github.com/wahern/luaossl/pull/199.

It is not a Linux nor macOS specific issue, but triggered whenever trying to compile this version of the module on any system with OpenSSL 1.1.1 while setting HAVE_EVP_KDF_CTX=1.

The root cause is this fix of ours, combined with this requirement from luaossl 20220711 and the fact that OpenSSL 1.1.1 does not include the core_names.h file (see openssl-1.1.1/includes vs openssl-3.0/includes).

I have tried a few things to workaround this issue without bumping to OpenSSL 3.x or having to rebuild the devel image in my local machine, but nothing worked given other constraints. I'll try a different approach tomorrow.

guicassolato avatar Nov 09 '22 18:11 guicassolato

Hi @imod,

Thanks to the great job from @guicassolato , the fix for aarch64 hosts was recently merged to master.

Let us know if you can setup the local environment

eguzki avatar Jan 20 '23 09:01 eguzki

Thanks @eguzki and @guicassolato - I just tried to setup my local environment and so far everything seems to be working fine!

imod avatar Feb 03 '23 16:02 imod