Älyverkko CLI application

Table of Contents

1. Introduction

The Älyverkko CLI application is a user-friendly tool developed in Java, specifically tailored to streamline the utilization of expansive language models through CPU-based computation in batch processing mode.

To illustrate its capabilities: Imagine harnessing the power of a vast language model, boasting approximately 100 billion parameters, solely relying on CPU computations and leveraging the open-source software llama.cpp. This setup requires a modern consumer-grade CPU and approximately 128 GB of RAM. To put this into perspective, 128 GB of RAM is financially comparable to purchasing a high-quality smartphone, making it an economical option for many users.

In contrast, executing the same model on an Nvidia GPU could result in costs that are significantly higher, potentially by an order of magnitude.

However, there is a trade-off: CPU-based processing for such extensive models is inherently slow. This means real-time interaction, like chatting with the AI, wouldn't be practical or enjoyable due to the lag in response times. Nevertheless, when deployed in a non-interactive batch mode, this "slow but smart" AI can complete numerous valuable tasks within a 24-hour window. For instance, it could generate a substantial amount of code, potentially exceeding what you could thoroughly review in the same timeframe. Additionally, it could process more documents than most individuals would be inclined to read manually.

The primary objective of the Älyverkko CLI project is to identify and enable applications where this "slow but smart" AI can excel. By utilizing llama.cpp as its inference engine, the project aims to unlock a variety of uses where batch processing is more advantageous than real-time interaction.

Here are some practical examples of tasks suited for the Älyverkko CLI application:

In summary, the Älyverkko CLI application opens up a realm of possibilities for leveraging powerful AI in scenarios where immediate responses are not critical, but high-quality batch processing output is highly beneficial.

Note: project is still in early stage.

1.1. Use cases

1.1.1. Domain: Natural Language Processing (NLP)

Problem Statement:

Analyze a set of customer reviews to determine overall sentiment and extract key features that customers appreciate or dislike about a product.

Usage Procedure:

  1. User collects customer reviews in plain text format within the configured mail directory. Lets say, about 150 kilobytes of reviews per input file (this is dictated by AI model available context size).
  2. Each review file is prefixed with "TOCOMPUTE:".
  3. The Älyverkko CLI application processes these files, generating sentiment analysis results and feature extraction insights.
  4. Results are appended to the original files in org-mode syntax, indicating AI responses.

1.1.2. Domain: Code Generation

Problem Statement:

Generate code snippets for a new software module based on detailed specifications provided by the developer.

Usage Procedure:

  1. Developer writes specifications in a text file within the mail directory, prefixed with "TOCOMPUTE:". Text file also contains relevant parts of the program source code and documentation. Älyverkko CLI joinfiles subcommand can be used to facilitate such text file preparation.
  2. The Älyverkko CLI application processes this file and generates the corresponding code snippets. The generated code is appended to the original specifications file, organized using org-mode syntax.
  3. Developer can review proposed changes and then integrate them back into original program source code.

Note: Most of the Älyverkko CLI program code is written in such a way by AI. This script is used to facilitate the process.

1.1.3. Domain: Content Creation

Problem Statement:

Draft an outline for a book on science fiction and improve its plot.

Usage Procedure:

  1. The book author writes a brief describing the outline of the plot and his book main idea for the novel.
  2. Älyverkko CLI processes this description and generates an outline with suggested headings and suggests possible improvements to the plot.

Here is example sci-fi book that was written with the help of Älyverkko CLI.

2. Installation

See Älyverkko CLI application installation tutorial.

3. Usage

3.1. Task preparation

The Älyverkko CLI application expects input files for processing in the form of plain text files within the specified mail directory (configured in the YAML configuration file).

Suggested usage flow is to prepare AI assignments within the Älyverkko CLI mail directory using normal text editor. Once AI assignment is ready for processing, you should initiate AI processing on that file.

3.1.1. "joinfiles" command

Note: See also alternative solution with similar goal: prelude.

The joinfiles command is a utility for aggregating the contents of multiple files into a single document, which can then be processed by AI. This is particularly useful for preparing comprehensive problem statements from various source files, such as software project directories or collections of text documents.

3.1.1.1. Usage

To use the joinfiles command, specify the source directory containing the files you wish to join and a topic name that will be used to generate the output file name: 

alyverkko-cli joinfiles -s /path/to/source/directory -t "my_topic"

If desired, you can also specify a glob pattern to match only certain files within the directory: 

alyverkko-cli joinfiles -s /path/to/source/directory -p "*.java" -t "my_topic"

After joining the files, you can choose to open the resulting document in text editor for further editing or review: 

alyverkko-cli joinfiles -t "my_topic" --edit
3.1.1.2. Options
  • -s, –src-dir: Specifies the source directory from which to join files.
  • -p, –pattern: An optional glob pattern to match specific files within the source directory.
  • -t, –topic: The topic name that will be used as a basis for the output file name and should reflect the subject matter of the joined content.
  • -e, –edit: Opens the joined file in text editor after the join operation is complete.
3.1.1.3. Example Use Case

Imagine you have a software project with various source files that you want to analyze using AI. You can use the joinfiles command to create a single document for processing: 

alyverkko-cli joinfiles -s /path/to/project -p "*.java" -t "software_analysis" --edit

This will recursively search the project directory for Java source files, aggregate their contents into a file named software_analysis.org (within AI processor input files directory), and open text editor on the file, so that you can add your analysis instructions or problem statement. Finally you Initiate AI processing and after some time, you will get results and the end of the file.

3.2. Initiate AI processing

Once your task file is prepared, you should place TOCOMPUTE: marker on the first line of that file, so that it will be considered for processing.

When the Älyverkko CLI detects a new or modified file in the mail directory:

  1. It checks if file has "TOCOMPUTE:" on the first line. If no, file is ignored. Otherwise Älyverkko CLI continues processing the file.
  2. It reads the content of the file and feeds it as an input for an AI model to generate a response.
  3. Once the AI has generated a response, the application appends this response to the original mail contents within the same file, using org-mode syntax to distinguish between the user's query and the assistant's reply. The updated file will contain both the original query (prefixed with: "* USER:*") and the AI's response (prefixed with "* ASSISTANT:"), ensuring a clear and organized conversation thread. "TOCOMPUTE:" is removed from the beginning of the file to avoid processing same file again.

Note: During AI task file preparation, feel free to save intermediary states as often as needed because AI engine will keep ignoring the file until TOCOMPUTE: line appears. Once AI assignment is ready, add 

TOCOMPUTE:

to the beginning of the file and save one last time. Älyverkko CLI will detect new task approximately within one second after file is saved and will start processing it.

If your text editor automatically reloads file when it was changed by other process in the filesystem, AI response will appear within text editor as soon as AI response is ready. If needed, you can add further queries at the end of the file and re-add "TOCOMPUTE:" at the beginning of the file. This way AI will process file again and file becomes stateful conversation. If you use GNU Emacs text editor, you can benefit from purpose-built GNU Emacs utilities.

3.3. Helpful GNU Emacs utilities

Note: GNU Emacs and following Emacs Lisp utilities are not required to use Älyverkko CLI. Their purpose is to increase comfort for existing GNU Emacs users.

3.3.1. Easily compose new problem statement for AI from emacs

The Elisp function ai-new-topic facilitates the creation and opening of a new Org-mode file dedicated to a user-defined topic within a specified directory. Now you can use this file within emacs to compose you problem statement to AI.

When ai-new-topic function triggered, it first prompts the user to input a topic name. This name will serve as the basis for the filename and the title within the document.

The function then constructs a file path by concatenating the pre-defined alyverkko-topic-files-directory (which should be set to your topics directory), the topic name, and the .org extension. If a file with this path does not already exist, the function will create a new file and open it for editing. 

(defvar alyverkko-topic-files-directory "/home/user/my-ai-mail-directory/"
  "Directory where topic files are stored.")

(defun alyverkko-new-topic ()
  "Create and open a topic file in the specified directory."
  (interactive)
  (let ((topic (read-string "Enter topic name: ")))
    (let ((file-path (concat alyverkko-topic-files-directory topic ".org")))
      (if (not (file-exists-p file-path))
          (with-temp-file file-path
            ))
      (find-file file-path)
      (goto-char (point-max))
      (org-mode))))

3.3.2. Easily signal to AI that problem statement is ready for solving

The function alyverkko-compute is designed to enhance the workflow of users working with the Älyverkko CLI application by automating the process of marking text files for computation with a specific AI model and prompt.

When function is invoked, it detects available prompts and allows user to select one.

Thereafter function detects available models from Älyverkko CLI configuration file and allows user to select one.

Finally function inserts at the beginning of currently opened file something like this: 

TOCOMPUTE: prompt=<user-chosen-prompt> model=<user-chosen-model>
  • Adjust prompt-dir variable to point to your prompts directory.
  • Adjust config-file variable to point to your Älyverkko CLI configuration file path.
(defun alyverkko-compute ()
  "Select a prompt and a model interactively, then insert them at the
   beginning of the current buffer."
  (interactive)
  (let ((prompt-dir "~/.config/alyverkko-cli/prompts/")
        (config-file "~/.config/alyverkko-cli/alyverkko-cli.yaml")
        (models '())
        )

    (with-temp-buffer
        (insert-file-contents config-file)
        ;; Move to the beginning of the prompts section
        (goto-char (point-min))
        (when (search-forward-regexp "^models:" nil t)
          ;; Collect all aliases
          (while (search-forward-regexp "^\\s-+- alias: \"\\([^\"]+\\)\"" nil t)
            (push (match-string 1) models))))

    (if (file-exists-p prompt-dir)
        (let* ((files (directory-files prompt-dir t "\\`[^.].*\\.txt\\'"))
               (aliases (mapcar (lambda (f) (file-name-base f)) files)))
          (if aliases
              (let ((selected-alias (completing-read "Select prompt alias: " aliases))
                    (model (completing-read "Select AI model: "              models)))
                (alyverkko-insert-tocompute-line selected-alias model))
            (message "No prompt files found.")))
      (message "Prompt directory not found."))))

(defun alyverkko-insert-tocompute-line (prompt-alias model)
  "Inserts TOCOMPUTE line with selected PROMPT-ALIAS and MODEL at the
   beginning of the buffer."
  (save-excursion
    (goto-char (point-min))
    (insert (format "TOCOMPUTE: prompt=%s model=%s\n" prompt-alias model))
    (save-buffer)))

4. Getting the source code

4.1. Source code

5. TODO

Ideas to be possibly implemented in the future:

5.1. Documentation

  • Add example problem statements and resulting solutions on various domains. Accompany them with precise usage procedure:
    • What user did, when, where and how.
    • How and where did result appear.
  • Add a section detailing the architecture and design decisions behind Älyverkko CLI.

5.2. System operation

  • Consider implementing a plugin architecture to allow third-party developers to extend Älyverkko CLI's functionality with custom modules or integrations.
  • Possibility to easily pause and resume Älyverkko CLI without loosing in-progress computation. Unix process stop and continue signals could possibly be used.
  • Explain how to monitor system performance and resource usage during AI processing tasks.
  • Introduce a comprehensive suite of automated tests to ensure the reliability and stability of new features before they are released.
  • Address potential security concerns, such as handling sensitive data in AI processing tasks and securing communication with external services.

5.3. Data management

  • in maildir ignore binary files, use joinfiles command as example how to ignore binary files. Perhaps extract plain text file detection into some utility class.
  • Make text editor configurable in application properties file.

5.4. Configuration and logging

  • Implement a fallback mechanism to use a system-wide configuration file located in `/etc/` if no user-specific configuration is found, enhancing out-of-the-box usability for new users.
  • Introduce optional logging of `llama.cpp` output to aid in debugging and performance monitoring without cluttering the standard output.
  • Implement a logging framework with configurable log levels to help users debug issues without cluttering the standard output.

5.5. Integration with external services

  • Add capabilities to connect with Jira, fetch content, and potentially update issues or comments based on AI processing results.
  • Implement similar integration with Confluence for content retrieval and updates.
  • Extend the application's reach by adding the ability to interact with arbitrary web sites, enabling information extraction and automation of web-based tasks.

5.6. Tooling enhancements

  • Incorporate Python script execution capabilities directly by the AI, expanding the range of available data manipulation and processing tools.
  • Integrate relational database access to leverage structured data storage and retrieval in workflows.
  • Enable web request functionality to interact with RESTful APIs or scrape web content as part of task execution.
  • Introduce a notebook feature that allows the AI to maintain and reference its own notes, fostering context retention across tasks.

5.7. Multimedia processing

  • Extend the application's capabilities to include voice capture and processing, opening up new avenues for interaction beyond text-based communication.
  • Implement image capture and processing features, enabling tasks that involve image analysis or content extraction from visual data.

5.8. Task preparation and queue management

  • Refactor the task queue mechanism to support:
    • Multiple task sources, including a REST API endpoint for programmatic task submission.
    • Load balancing across multiple executors (instances of Älyverkko CLI) with dynamic registration and unregistration without system interruption.
    • Task priority assignments, ensuring that critical tasks are processed in a timely manner.
  • Offer guidance on preparing input files for batch processing, including best practices for formatting data and structuring problem statements.

5.9. User interface development

  • Create a web-based UI to provide users with an interface for task submission and result retrieval, improving accessibility and user experience.
  • Integrate Quality of Service (QoS) concepts within the UI to ensure equitable resource allocation among users.
  • Implement administrative features for managing user accounts and system resources, maintaining a secure and efficient operating environment.

Created: 2024-10-01 ti 01:07

Validate