Refactor pipeline downloads

[jedwards1230]

What does this do?

This pull request focuses on improving logging, refactoring initial pipeline downloads, and adding new configuration options. The most important changes include the addition of new environment variables and refactoring of the start.sh and main.py files to improve readability and functionality.

I've mostly been tweaking here and there. I can separate this into two PRs if that's preferred, one for the logging, then one for the download refactoring. Happy to make any changes!

Configuration Changes:

• Added new environment variables LOG_LEVEL, RESET_PIPELINES_DIR, PIPELINES_REQUIREMENTS_PATH, PIPELINES_URLS, and SUPPRESS_PIP_OUTPUT in config.py.

Code Refactoring:

• Removed initial pipeline download logic from start.sh and wrote it to utils/pipelines/downloads.py.
• Refactored main.py to improve readability and functionality by reorganizing imports, removing unused imports, and modularizing functions related to downloading and installing pipelines.

Logging Improvements:

• Replaced print statements with appropriate logging levels (debug, info, warning, error) to improve log management.

Example Log Output from https://github.com/jedwards1230/lilbro-pipelines/tree/main/pipelines

pipelines  | INFO:     Started server process [7]
pipelines  | INFO:     Waiting for application startup.
pipelines  | INFO (utils.pipelines.downloads): Resetting pipelines directory: ./pipelines
pipelines  | DEBUG (utils.pipelines.downloads): All contents in ./pipelines have been removed.
pipelines  | INFO (utils.pipelines.downloads): ./pipelines has been recreated.
pipelines  | DEBUG (utils.pipelines.downloads): PIPELINES_REQUIREMENTS_PATH not specified or file not found. Skipping installation of requirements.
pipelines  | INFO (utils.pipelines.downloads): Downloading pipeline files from https://github.com/jedwards1230/lilbro-pipelines/tree/main/pipelines to ./pipelines...
pipelines  | DEBUG (utils.pipelines.downloads): Found folder from GitHub...
pipelines  | DEBUG (utils.pipelines.downloads): Downloading folder from GitHub: https://github.com/jedwards1230/lilbro-pipelines/tree/main/pipelines
pipelines  | DEBUG (utils.pipelines.downloads): Downloaded folder pipelines from https://github.com/jedwards1230/lilbro-pipelines/tree/main/pipelines
pipelines  | DEBUG (main): Created subfolder: ./pipelines/ollama_manifold_pipeline
pipelines  | DEBUG (main): Created valves.json in: ./pipelines/ollama_manifold_pipeline
pipelines  | INFO (utils.pipelines.downloads): No frontmatter found in ./pipelines/ollama_manifold_pipeline.py.
pipelines  | INFO (main): Loaded module: ollama_manifold_pipeline
pipelines  | DEBUG (main): Updated valves for module: ollama_manifold_pipeline
pipelines  | INFO (main): Loaded module: ollama_manifold_pipeline
pipelines  | DEBUG (main): Created subfolder: ./pipelines/anthropic_manifold_pipeline
pipelines  | DEBUG (main): Created valves.json in: ./pipelines/anthropic_manifold_pipeline
pipelines  | INFO (utils.pipelines.downloads): Installing packages: ['requests', 'sseclient-py']
pipelines  | INFO (main): Loaded module: anthropic_manifold_pipeline
pipelines  | DEBUG (main): Updated valves for module: anthropic_manifold_pipeline
pipelines  | INFO (main): Loaded module: anthropic_manifold_pipeline
pipelines  | DEBUG (main): Created subfolder: ./pipelines/google_manifold_pipeline
pipelines  | DEBUG (main): Created valves.json in: ./pipelines/google_manifold_pipeline
pipelines  | INFO (utils.pipelines.downloads): Installing packages: ['google-generativeai']
pipelines  | INFO (main): Loaded module: google_manifold_pipeline
pipelines  | DEBUG (main): Updated valves for module: google_manifold_pipeline
pipelines  | INFO (main): Loaded module: google_manifold_pipeline
pipelines  | DEBUG (main): Created subfolder: ./pipelines/openai_manifold_pipeline
pipelines  | DEBUG (main): Created valves.json in: ./pipelines/openai_manifold_pipeline
pipelines  | INFO (utils.pipelines.downloads): Installing packages: ['requests', 'boto3', 'openai']
pipelines  | INFO (main): Loaded module: openai_manifold_pipeline
pipelines  | DEBUG (main): Updated valves for module: openai_manifold_pipeline
pipelines  | INFO (main): Loaded module: openai_manifold_pipeline
pipelines  | DEBUG (main): Created subfolder: ./pipelines/agent_test_pipeline
pipelines  | DEBUG (main): Created valves.json in: ./pipelines/agent_test_pipeline
pipelines  | INFO (utils.pipelines.downloads): Installing packages: ['git+https://github.com/openai/swarm.git']
pipelines  | INFO (main): Loaded module: agent_test_pipeline
pipelines  | DEBUG (main): Updated valves for module: agent_test_pipeline
pipelines  | INFO (main): Loaded module: agent_test_pipeline
pipelines  | DEBUG (main): Created subfolder: ./pipelines/openai_dalle_manifold_pipeline
pipelines  | DEBUG (main): Created valves.json in: ./pipelines/openai_dalle_manifold_pipeline
pipelines  | INFO (utils.pipelines.downloads): No requirements found in frontmatter of ./pipelines/openai_dalle_manifold_pipeline.py.
pipelines  | INFO (main): Loaded module: openai_dalle_manifold_pipeline
pipelines  | DEBUG (main): Updated valves for module: openai_dalle_manifold_pipeline
pipelines  | INFO (main): Loaded module: openai_dalle_manifold_pipeline
pipelines  | DEBUG (main): Created subfolder: ./pipelines/aws_bedrock_claude_pipeline
pipelines  | DEBUG (main): Created valves.json in: ./pipelines/aws_bedrock_claude_pipeline
pipelines  | INFO (utils.pipelines.downloads): Installing packages: ['requests', 'boto3']
pipelines  | INFO (main): Loaded module: aws_bedrock_claude_pipeline
pipelines  | DEBUG (main): Updated valves for module: aws_bedrock_claude_pipeline
pipelines  | INFO (main): Loaded module: aws_bedrock_claude_pipeline
pipelines  | INFO:     Application startup complete.
pipelines  | INFO:     Uvicorn running on http://0.0.0.0:9099 (Press CTRL+C to quit)

Dependency Updates:

• Added gitpython to requirements-minimum.txt and requirements.txt to support pipeline downloading from GitHub repositories.
• Added environment variable SUPPRESS_PIP_OUTPUT to suppress pip install output.
• Added the ability to pass in many packages to the pip install command at once.

What is left?

• [ ] Tests for various pipeline URLs