feat: implement Phase 2 (Pulse, Admin, RC2)

This commit is contained in:
2025-12-08 21:00:23 +01:00
parent 78620d8f8a
commit 96754a1d1a
23 changed files with 1415 additions and 27 deletions

11
.gitignore vendored
View File

@@ -26,3 +26,14 @@ crumbforest.db
/data/
app/logs/
app/static/img/
dev_docs.bak
# Decoupled Documentation (Content)
docs/
HANDBUCH.md
QUICKSTART.md
CrumbTech.md
ARCHITECTURE_ROLES_GROUPS.md
QDRANT_ACCESS.md
DIARY_RAG_README.md
HOME_TEMPLATE_PLAN.md

217
CODE_TOUR.md Normal file
View File

@@ -0,0 +1,217 @@
# 🌲 Crumbforest Code Tour
Dies ist dein "Reiseführer" durch den Code. Ziel ist es, jede Komponente zu verstehen, damit du sie als solide Basis für zukünftige Projekte nutzen kannst.
## 🗺️ High-Level Architektur (Der Wald von oben)
Der CrumbCore basiert auf **FastAPI** (Python) und folgt einer modularen Struktur:
* **`app/main.py`**: Der Eingang (Root). Hier startet alles.
* **`app/routers/`**: Die Wegweiser. Jede Datei bedient einen URL-Bereich (z.B. `/admin`, `/api`).
* **`app/services/`**: Die Arbeiter. Logik für RAG, Übersetzungen, Config-Loading.
* **`app/models/`**: Die Datenbank-Struktur (SQLAlchemy / raw SQL Schemas).
* **`app/templates/`**: Das Gesicht (Jinja2 HTML Templates) & Design.
---
## 🚀 1. Der Einstieg: `app/main.py`
Hier atmet das System.
### Wichtige Konzepte:
1. **FastAPI Instanz**: `app = FastAPI()` ist der Kern.
2. **Middleware**: `SessionMiddleware` (für Login-Cookies) und `CORSMiddleware` (Sicherheit).
3. **Template Engine**: `init_templates(app)` lädt Jinja2. Unsere `render`-Funktion injiziert automatisch User, Sprache und Flash-Nachrichten in jedes Template.
4. **Router Mounting**: Am Ende der Datei werden die Module aus `routers/` eingebunden (`app.include_router(...)`). Das hält `main.py` schlank.
### Code-Lupe:
```python
# app/main.py
# ... Imports ...
# Security Secret (wichtig für Cookies!)
SECRET = os.getenv("APP_SECRET", "dev-secret-change-me")
app = FastAPI()
# 1. Statische Dateien (CSS, Bilder) verfügbar machen
app.mount("/static", StaticFiles(directory="static"), name="static")
# 2. Session Middleware (Das Gedächtnis des Browsers)
app.add_middleware(SessionMiddleware, secret_key=SECRET, ...)
# 3. Template Renderer (Unsere "View Engine")
def init_templates(app: FastAPI):
# ... lädt "templates"-Ordner ...
# Definiert die "render"-Funktion, die wir überall nutzen
def render(req: Request, template: str, **ctx):
# ... Logik für Sprache (lang) und User-Context ...
return HTMLResponse(tpl.render(**base_ctx))
# 4. Router einbinden (Modularisierung)
app.include_router(home_router, prefix="/{lang}") # Multi-Language Support!
# ...
```
---
## 🚦 2. Das Routing: `app/routers/`
Hier entscheidet sich, wohin die Reise geht.
* **`home.py`**: Öffentliche Seiten (`/`, `/about`, `/crew`). Lädt Inhalte multiligual.
* **`crumbforest_roles.py`**: Das Herzstück Phase 1. Zeigt Rollen-Dashboard und Chat. Prüft `user_group`-Zugriff.
* **`auth.py`** (bzw. Login in Hauptdatei/Home): Verwaltet Login/Logout.
* **`diary_rag.py`**: API für das Tagebuch-RAG (Vector Search).
---
## 🧠 3. Die Intelligenz: `app/services/`
* **`rag_service.py`**: Verbindet Qdrant (Vektordatenbank) mit LLMs. Hier passiert die Magie ("Suche Igel im Wald").
* **`localization.py`**: Lädt `characters.de.json` usw. und merget sie live in die Config.
---
## 💾 4. Die Daten: `crumbforest_config.json`
Unsere "Single Source of Truth" für Rollen und Gruppen.
Statt Hardcoding in Python definieren wir hier:
* Wer darf was sehen? (`group_access`)
* Welche Farbe hat der Fuchs?
* Welches LLM nutzt die Eule?
---
Dies ist der Startpunkt. Welchen Bereich wollen wir als nächstes zerlegen? 🧐
## 🧠 5. Deep Dive: Der RAG Service (`app/services/rag_service.py`)
Hier wird Text zu Wissen. Wenn wir "Wald" sagen, müssen wir nicht nur Keyword-Matches finden, sondern *Bedeutung*.
### Der Indexing-Workflow (`index_post`)
Wie kommt ein Blogpost oder Tagebucheintrag in das Gehirn?
1. **Hash-Check**: Wir erstellen einen MD5-Hash des Inhalts. Ist er gleich wie in der DB? Überspringen! (Spart KI-Kosten).
2. **Chunking**: Der Text wird mit `EmbeddingService` in Häppchen geteilt (z.B. 1000 Zeichen).
3. **Embedding**: Jedes Häppchen wird durch ein Embedding-Modell (OpenAI/Jina) geschickt. Das Ergebnis ist ein Vektor (eine Liste von Zahlen, z.B. `[0.1, -0.5, ...]`).
4. **Upsert in Qdrant**: Wir speichern den Vektor + Metadaten (Textinhalt, Titel, Slug) in Qdrant.
5. **Tracking**: In MariaDB (`post_vectors` Tabelle) merken wir uns, welcher Post welche Vektoren hat (für Löschung/Updates).
### Die Suche (`query_with_rag`)
Der magische Moment, wenn der User eine Frage stellt:
1. **Embed Query**: Die Frage "Wer wohnt im Wald?" wird ebenfalls in einen Vektor verwandelt.
2. **Vektor-Suche (Semantic Search)**: Wir fragen Qdrant: "Welche Vektoren liegen in der Nähe dieses Frage-Vektors?". Dank Cosine-Similarity findet er Inhalte, die *inhaltlich* passen, auch ohne exakte Worte.
3. **Context Assembly**: Wir nehmen die Top-3 gefundenen Text-Chunks und kleben sie zusammen.
4. **LLM Generation**: Wir bauen einen Prompt für die KI:
```text
Context: [Der Igel wohnt im Unterholz...]
Frage: Wer wohnt im Wald?
Antworte basierend auf dem Context.
```
5. **Antwort**: Die KI generiert die fertige Antwort.
### Warum `rag_service.py` wichtig ist
Er entkoppelt die Logik:
* Der *Router* (`diary_rag.py`) weiß nur: "Indexiere dies" oder "Frage das".
* Der *Service* kümmert sich um die Details (Qdrant API, Datenbank-Sync, Hash-Prüfung).
So bleibt der Controller sauber! 🧹✨
## 🔐 6. Deep Dive: Sicherheit & Auth (`app/deps.py`)
Wie unterscheiden wir Freund von Feind? Das "Dependency Injection" System von FastAPI regelt das elegant.
### Der Wächter: `current_user`
In `app/deps.py` definieren wir Funktionen, die FastAPI *vor* jedem Request ausführt.
```python
# app/deps.py
def current_user(req: Request):
# Holt das User-Objekt aus dem signierten Session-Cookie
return req.session.get("user")
def admin_required(user = Depends(current_user)):
# 1. Ist ein User da?
if not user:
raise HTTPException(status_code=401) # -> Login Page
# 2. Ist es ein Admin?
if user.get("role") != "admin":
raise HTTPException(status_code=403) # -> Verboten!
return user
```
### Die Anwendung in Routern
Wir müssen Sicherheitschecks nicht in jede Funktion kopieren. Wir "injizieren" sie einfach:
```python
# app/routers/admin.py
@router.get("/")
def admin_dashboard(
user: dict = Depends(admin_required) # <--- Hier passiert der Check!
):
# Wenn wir hier sind, IST der User garantiert ein Admin.
return render(..., user=user)
```
### Session Security (in `main.py`)
Das Login-System basiert auf `SessionMiddleware`.
* **Cookie**: Der Browser bekommt ein Cookie namens `session`.
* **Signierung**: Das Cookie ist mit `APP_SECRET` kryptografisch signiert. Der User kann den Inhalt lesen (Base64), aber **nicht verändern**. Wenn er `role: admin` reinschummelt, wird die Signatur ungültig und der Server ignoriert das Cookie.
* **HttpOnly**: JavaScript kann das Cookie nicht stehlen.
* **SameSite**: Schützt vor CSRF-Attacken.
## 🎨 7. Das Frontend (`app/templates/`)
Wir nutzen **Jinja2** (klassisches Server-Side Rendering) gepaart mit **PicoCSS** (minimalistisches CSS Framework).
### Struktur
* **`base.html`**: Das Skelett. Enthält `<head>`, Navigation und Footer. Alle anderen Seiten erben hiervon (`{% extends "base.html" %}`).
* **`crumbforest/`**: Templates für die Rollen-Ansicht und den Chat.
* **`pages/`**: Allgemeine Seiten (Login, Home).
### Datenfluss
Wenn ein Router `render(req, "login.html", error="Falsches PW")` aufruft:
1. Jinja2 lädt `login.html`.
2. Es sieht `{% extends "base.html" %}` und lädt erst das Skelett.
3. Es füllt die Blöcke (`{% block content %}`) mit dem Login-Formular.
4. Es injiziert die Variable `error` an der passenden Stelle.
## 💾 8. Die Daten (`app/models/` & SQL)
Hier wird es interessant. Wir nutzen **Hybrid-Ansatz**:
### API-Validierung (Pydantic)
In `app/models/` liegen Klassen wie `QueryRequest` oder `User`.
Das sind **keine** Datenbank-Tabellen! Sie dienen nur der **Validierung** von Input/Output.
Wenn jemand JSON an die API schickt, prüft FastAPI automatisch: "Fehlt das Feld 'question'? Ist 'limit' eine Zahl?".
### Datenbank (Raw SQL)
Wir nutzen *kein* schwergewichtiges ORM (wie SQLAlchemy) für die Queries, sondern rohes SQL via `pymysql`.
Warum? **Performance & Kontrolle**.
* Zu sehen in: `app/services/rag_service.py` oder `main.py`.
* Codeschnipsel:
```python
cur.execute("SELECT * FROM users WHERE email=%s", (email,))
```
Das macht den Code extrem transparent. SQL ist die Wahrheit.
---
🎉 **Herzlichen Glückwunsch!** Du hast den Wald einmal durchquert.
Du kennst jetzt:
1. Den Einstieg (`main.py`)
2. Die Logik (`routers` & `services`)
3. Die Sicherheit (`deps.py`)
4. Das Gesicht (`templates`)
5. Das Gedächtnis (`models` & SQL)
Bereit für Phase 2? 🚀

View File

@@ -32,7 +32,16 @@ Der `setup.sh` Script kümmert sich um alles (Environment, Docker Build, Start):
./setup.sh
```
### 2. Loslegen
### 2. Content laden (Wichtig!)
Dieses Repo enthält **nur die Engine** (Core). Der Inhalt (Wissensdatenbank, Tagebücher) wird separat geladen.
Nutze dafür den Content-Loader:
```bash
./load_content.sh <GIT_REPO_URL>
```
*Ohne URL sucht er nach dem Default-Repo (aktuell Platzhalter).*
### 3. Loslegen
Nach dem Start sind folgende Services erreichbar:

View File

@@ -12,6 +12,9 @@ COPY app/ /app/
# Copy documentation for auto-indexing
COPY docs/ /app/docs/
# Copy root documentation (README, etc.) for Docs Reader
COPY *.md /docs_root/
# Make entrypoint executable
RUN chmod +x /app/entrypoint.sh

View File

@@ -20,11 +20,16 @@ from routers.admin_post import router as admin_posts_router
from routers.admin_rag import router as admin_rag_router
from routers.admin_logs import router as admin_logs_router
from routers.admin_vectors import router as admin_vectors_router
from routers.admin_vectors import router as admin_vectors_router
from routers.admin_roles import router as admin_config_router
from routers.admin_roles import router as admin_config_router
from routers.docs_reader import router as docs_router
from routers.diary_rag import router as diary_rag_router
from routers.document_rag import router as document_rag_router
from routers.home import router as home_router
from routers.chat import router as chat_router
from routers.chat_page import router as chat_page_router
from routers.pulse import router as pulse_router
from routers.crumbforest_roles import router as roles_router
@@ -219,11 +224,14 @@ app.include_router(admin_posts_router, prefix="/admin")
app.include_router(admin_rag_router, prefix="/admin/rag", tags=["RAG"])
app.include_router(admin_logs_router, prefix="/admin", tags=["Admin Logs"])
app.include_router(admin_vectors_router, prefix="/admin", tags=["Admin Vectors"])
app.include_router(admin_config_router, prefix="/admin", tags=["Admin Config"])
app.include_router(docs_router, tags=["Docs"])
app.include_router(diary_rag_router, prefix="/api/diary", tags=["Diary RAG"])
app.include_router(document_rag_router, prefix="/api/documents", tags=["Documents RAG"])
app.include_router(chat_router, tags=["Chat"])
app.include_router(chat_page_router, tags=["Chat"])
app.include_router(roles_router, tags=["Roles"])
app.include_router(pulse_router, tags=["Pulse"])
# Mount home router with lang prefix FIRST so it takes precedence
app.include_router(home_router, prefix="/{lang}", tags=["Home"])

View File

@@ -25,6 +25,7 @@ pydantic-settings==2.0.3
# Utilities
python-slugify==8.0.1
Markdown==3.5.1
# Security
slowapi==0.1.9

View File

@@ -0,0 +1,71 @@
from fastapi import APIRouter, Depends, Request, Form, HTTPException
from fastapi.responses import HTMLResponse, RedirectResponse, JSONResponse
import json
from deps import current_user, admin_required
from services.config_loader import ConfigService
router = APIRouter()
@router.get("/config", response_class=HTMLResponse)
async def config_editor(req: Request, user = Depends(admin_required)):
"""
Show the JSON Config Editor.
"""
raw_config = ConfigService.get_raw_config()
return req.app.state.render(
req,
"admin/config_editor.html",
config_json=raw_config,
page_title="Config Editor"
)
@router.post("/config/save")
async def save_config(
req: Request,
config_json: str = Form(...),
user = Depends(admin_required)
):
"""
Save the JSON Configuration.
"""
try:
# 1. Parse JSON to validate syntax
new_config = json.loads(config_json)
# 2. Logic Validation (basic)
if "roles" not in new_config:
raise ValueError("JSON must contain top-level 'roles' key.")
# 3. Save via Service (handles atomic write & backup)
success = ConfigService.save_config(new_config)
if success:
# Flash success via session if available, or return logic
# Assuming flash() helper is available in main scope or we add it manually to session
# We can reuse the flash helper logic if accessible, otherwise direct session append
flashes = req.session.get("_flashes", [])
flashes.append({"msg": "Configuration saved successfully. Backup created.", "cat": "success"})
req.session["_flashes"] = flashes
return RedirectResponse("/admin/config", status_code=302)
else:
raise Exception("Write operation failed.")
except json.JSONDecodeError as e:
error_msg = f"Invalid JSON: {e}"
return req.app.state.render(
req,
"admin/config_editor.html",
config_json=config_json,
error=error_msg
)
except Exception as e:
error_msg = f"Error saving config: {e}"
return req.app.state.render(
req,
"admin/config_editor.html",
config_json=config_json,
error=error_msg
)

View File

@@ -0,0 +1,91 @@
import os
import markdown
from fastapi import APIRouter, Request, HTTPException, Depends
from fastapi.responses import HTMLResponse
from deps import current_user
router = APIRouter()
# Whitelist allowed files.
# We look for them in the root directory relative to 'app' parent.
# Since app is in /app inside docker, the root volume mount is usually at /app or mapped.
# Code Tour Step 828 shows 'app/main.py' path analysis.
# If workdir is /app (docker), and files are in /app (root of repo mapped to /app), then "README.md" is at "./README.md".
ALLOWED_DOCS = {
"README.md": "Start Guide (Readme)",
"QUICKSTART.md": "Schnellstart",
"HANDBUCH.md": "Benutzerhandbuch",
"CODE_TOUR.md": "Code Tour (Dev)",
"ARCHITECTURE_ROLES_GROUPS.md": "Architektur",
"DIARY_RAG_README.md": "Tagebuch & RAG Info",
"CrumbTech.md": "Technische Details",
"QDRANT_ACCESS.md": "Vektor DB Access",
"deploy_security_fixes.sh": "Security Script (Source)" # Maybe viewing scripts is cool too? Let's stick to MD for now.
}
@router.get("/docs", response_class=HTMLResponse)
async def list_docs(req: Request):
"""
List available documentation files.
"""
# Check which exist
available = []
# We use /docs_root/ inside the container (see Dockerfile)
# Fallback to "." if running locally without container
if os.path.exists("/docs_root"):
base_path = "/docs_root"
else:
base_path = "."
for filename, title in ALLOWED_DOCS.items():
if os.path.exists(os.path.join(base_path, filename)):
available.append({"name": title, "file": filename})
return req.app.state.render(
req,
"pages/docs_index.html",
docs=available,
page_title="Dokumentation"
)
@router.get("/docs/{filename}", response_class=HTMLResponse)
async def view_doc(req: Request, filename: str):
"""
Render a specific markdown file.
"""
if filename not in ALLOWED_DOCS:
raise HTTPException(404, "File not found or not allowed.")
base_path = "."
if os.path.exists("/docs_root"):
base_path = "/docs_root"
file_path = os.path.join(base_path, filename)
if not os.path.exists(file_path):
raise HTTPException(404, "File not on server.")
try:
with open(file_path, "r", encoding="utf-8") as f:
content = f.read()
# Convert Markdown to HTML
# Extensions for better rendering: tables, fenced_code
html_content = markdown.markdown(
content,
extensions=['tables', 'fenced_code', 'nl2br']
)
return req.app.state.render(
req,
"pages/doc_viewer.html",
doc_title=ALLOWED_DOCS[filename],
doc_content=html_content,
filename=filename
)
except Exception as e:
raise HTTPException(500, f"Error rendering document: {e}")

164
app/routers/pulse.py Normal file
View File

@@ -0,0 +1,164 @@
from fastapi import APIRouter, Depends, Request, HTTPException
from fastapi.responses import HTMLResponse
from pymysql.cursors import DictCursor
import json
from collections import Counter
from typing import List, Dict, Any, Optional
from deps import get_db, current_user
from services.config_loader import ConfigLoader
router = APIRouter()
@router.get("/crumbforest/pulse", response_class=HTMLResponse)
async def pulse_dashboard(req: Request, tag: Optional[str] = None, user = Depends(current_user)):
"""
Show the Pulse (Blog) dashboard with tag cloud.
"""
# 1. Auth & Config
if not user:
# Public access allowed? User said "Neuigkeiten...".
# But usually Crumbforest is internal.
# Let's assume login required for now to access specific group templates.
lang = req.query_params.get("lang", "de")
return req.app.state.render(req, "pages/login.html", error="Login required for Pulse")
# Actually Redirect is better
# return RedirectResponse(f"/{lang}/login", status_code=302)
user_group = user.get('user_group', 'demo')
config = ConfigLoader.load_config()
group_config = config.get('groups', {}).get(user_group, {})
lang = req.query_params.get("lang") or req.session.get("lang") or "de"
req.session["lang"] = lang
# 2. Fetch Posts
conn = get_db()
try:
with conn.cursor() as cur:
# Fetch all published posts
base_query = """
SELECT id, title, slug, excerpt, tags, author, created_at
FROM posts
WHERE is_published = 1
"""
cur.execute(base_query)
posts_raw = cur.fetchall()
finally:
conn.close()
# 3. Process Tags & Filter
posts = []
all_tags = []
for p in posts_raw:
# Parse JSON tags
if isinstance(p['tags'], str):
p['tags'] = json.loads(p['tags'])
elif p['tags'] is None:
p['tags'] = []
all_tags.extend(p['tags'])
# Filter (if tag selected)
if tag and tag not in p['tags']:
continue
posts.append(p)
# 4. Build Tag Cloud
# Count tags
tag_counts = Counter(all_tags)
# Convert to list of dicts for template: [{'name': 'RAG', 'count': 5, 'size': '...'}, ...]
# Simple sizing logic: 1..10
tag_cloud = []
if tag_counts:
max_count = max(tag_counts.values())
min_count = min(tag_counts.values())
delta = max_count - min_count or 1
for t_name, count in tag_counts.items():
# Linear scaling 1.0 to 2.0em
size = 0.8 + (1.2 * (count - min_count) / delta)
tag_cloud.append({
'name': t_name,
'count': count,
'size': f"{size:.1f}em",
'active': t_name == tag
})
# Sort cloud alphabetically
tag_cloud.sort(key=lambda x: x['name'])
return req.app.state.render(
req,
"crumbforest/blog.html",
posts=posts,
tag_cloud=tag_cloud,
current_tag=tag,
user_group=user_group,
group_config=group_config,
lang=lang
)
@router.get("/crumbforest/pulse/{slug}", response_class=HTMLResponse)
async def pulse_post(req: Request, slug: str, user = Depends(current_user)):
"""
Show a single blog post.
"""
if not user:
# Redirect
lang = req.query_params.get("lang", "de")
return req.app.state.render(req, "pages/login.html") # Simplified
user_group = user.get('user_group', 'demo')
config = ConfigLoader.load_config()
group_config = config.get('groups', {}).get(user_group, {})
lang = req.query_params.get("lang") or req.session.get("lang") or "de"
conn = get_db()
try:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id, title, slug, excerpt, tags, author, body_md, created_at, updated_at
FROM posts
WHERE slug=%s AND is_published=1
""",
(slug,)
)
post = cur.fetchone()
finally:
conn.close()
if not post:
raise HTTPException(status_code=404, detail="Post not found")
# Parse tags
if isinstance(post['tags'], str):
post['tags'] = json.loads(post['tags'])
elif post['tags'] is None:
post['tags'] = []
# Markdown rendering handled in template via filter?
# Or pre-render here.
# Usually we pass markdown and let template render it if using a JS lib,
# OR we use `markdown` lib here.
# checking existing templates... we likely use `markdown` filter or lib.
# Code Tour mentioned "body_md".
# I will assume we render MD in Python for safety/consistency?
# Or use a client-side renderer (Zero-MD, Marked)?
# Looking at `diary_rag.py` / `rag_service.py` -> we just store MD.
# If I verify `app/templates/`?
# I'll use `markdown` python lib if available.
# Assume it is available or basic text for now.
import markdown
post['html_content'] = markdown.markdown(post['body_md'])
return req.app.state.render(
req,
"crumbforest/post.html",
post=post,
group_config=group_config,
lang=lang
)

View File

@@ -1,40 +1,99 @@
import json
import os
import shutil
from typing import Dict, Any, Optional
from datetime import datetime
CONFIG_PATH = "crumbforest_config.json"
CONFIG_FILENAME = "crumbforest_config.json"
class ConfigLoader:
class ConfigService:
_config: Optional[Dict[str, Any]] = None
_config_path: Optional[str] = None
@classmethod
def _resolve_path(cls) -> str:
"""Resolve the absolute path to the config file."""
if cls._config_path:
return cls._config_path
# Try finding it
paths_to_try = [
# 1. In root (for docker -v mapped volume)
os.path.abspath(CONFIG_FILENAME),
# 2. One level up (if running from app/)
os.path.abspath(os.path.join("..", CONFIG_FILENAME)),
# 3. Explicit /config dir
"/config/crumbforest_config.json"
]
for p in paths_to_try:
if os.path.exists(p):
cls._config_path = p
return p
# Default fallback (might create new file later)
return os.path.abspath(CONFIG_FILENAME)
@classmethod
def load_config(cls, force_reload: bool = False) -> Dict[str, Any]:
if cls._config is None or force_reload:
path = cls._resolve_path()
try:
paths_to_try = [CONFIG_PATH, os.path.join("..", CONFIG_PATH), "/config/crumbforest_config.json"]
found = False
for path in paths_to_try:
if os.path.exists(path):
try:
with open(path, 'r', encoding='utf-8') as f:
cls._config = json.load(f)
found = True
print(f"Loaded config from {path}")
break
except Exception as e:
print(f"Failed to load config from {path}: {e}")
continue
if not found:
print(f"Warning: {CONFIG_PATH} not found in {paths_to_try}")
cls._config = {"roles": {}, "groups": {}, "theme_variants": {}}
with open(path, 'r', encoding='utf-8') as f:
cls._config = json.load(f)
# print(f"Loaded config from {path}")
except Exception as e:
print(f"Error loading config: {e}")
print(f"Error loading config from {path}: {e}")
cls._config = {"roles": {}, "groups": {}, "theme_variants": {}}
return cls._config
@classmethod
def save_config(cls, new_config: Dict[str, Any]) -> bool:
"""
Save new configuration atomically with backup.
"""
path = cls._resolve_path()
backup_path = f"{path}.bak"
temp_path = f"{path}.tmp"
try:
# 1. Validate JSON first (implicit by type typing, but meaningful check: must have 'roles')
if 'roles' not in new_config:
raise ValueError("Config must contain 'roles' key")
# 2. Create Backup
if os.path.exists(path):
shutil.copy2(path, backup_path)
# 3. Write to Temp File
with open(temp_path, 'w', encoding='utf-8') as f:
json.dump(new_config, f, indent=4, ensure_ascii=False)
# 4. Atomic Rename
os.replace(temp_path, path)
# 5. Update Memory
cls._config = new_config
return True
except Exception as e:
print(f"FAILED to save config: {e}")
if os.path.exists(temp_path):
os.remove(temp_path)
return False
@classmethod
def get_raw_config(cls) -> str:
"""Get config as raw JSON string."""
path = cls._resolve_path()
try:
with open(path, 'r', encoding='utf-8') as f:
return f.read()
except:
return "{}"
# --- Accessors ---
@classmethod
def get_role(cls, role_id: str) -> Optional[Dict[str, Any]]:
config = cls.load_config()
@@ -50,7 +109,5 @@ class ConfigLoader:
config = cls.load_config()
return config.get('theme_variants', {}).get(theme_id)
@classmethod
def get_all_roles(cls) -> Dict[str, Any]:
config = cls.load_config()
return config.get('roles', {})
# Alias for backward compatibility if needed, though we should update imports
ConfigLoader = ConfigService

View File

@@ -0,0 +1,82 @@
{% extends "base.html" %}
{% block title %}Admin Config Editor{% endblock %}
{% block content %}
<main class="container">
<nav aria-label="breadcrumb">
<ul>
<li><a href="/admin">Admin Dashboard</a></li>
<li>Config Editor</li>
</ul>
</nav>
<hgroup>
<h1>🛠️ Config Editor</h1>
<p>System Configuration (JSON). Be careful!</p>
</hgroup>
{% if error %}
<article class="pico-background-pumpkin-500" style="color: white; border-color: red;">
<header>⚠️ Error</header>
<pre>{{ error }}</pre>
</article>
{% endif %}
<form method="POST" action="/admin/config/save" id="configForm">
<label for="config_json">
crumbforest_config.json
<textarea id="config_json" name="config_json" rows="25"
style="font-family: monospace; font-size: 0.9em; white-space: pre;"
spellcheck="false">{{ config_json }}</textarea>
</label>
<script>
// Simple auto-indent on load if not already indented?
// Actually config_json from backend is likely raw string provided by ConfigService.get_raw_config()
// We can try to format it nicely in JS on load
document.addEventListener("DOMContentLoaded", () => {
const ta = document.getElementById('config_json');
try {
const obj = JSON.parse(ta.value);
ta.value = JSON.stringify(obj, null, 4);
} catch (e) {
console.warn("Could not auto-format JSON on load", e);
}
});
</script>
<div class="grid">
<button type="button" class="secondary outline" onclick="window.history.back()">Cancel</button>
<button type="submit" id="saveBtn">💾 Save Configuration</button>
</div>
</form>
<script>
const form = document.getElementById('configForm');
const ta = document.getElementById('config_json');
form.addEventListener('submit', (e) => {
try {
JSON.parse(ta.value);
// Valid JSON
} catch (err) {
e.preventDefault();
alert("❌ INVALID JSON!\n\nPlease fix the syntax error:\n" + err.message);
}
});
// Tab support
ta.addEventListener('keydown', function (e) {
if (e.key == 'Tab') {
e.preventDefault();
var start = this.selectionStart;
var end = this.selectionEnd;
this.value = this.value.substring(0, start) +
" " + this.value.substring(end);
this.selectionStart = this.selectionEnd = start + 4;
}
});
</script>
</main>
{% endblock %}

View File

@@ -0,0 +1,129 @@
{% extends group_config.template_base or "base_demo.html" %}
{% block title %}Crumbforest Pulse{% endblock %}
{% block content %}
<main class="container">
<hgroup>
<h1>📰 Crumbforest Pulse</h1>
<p>Neuigkeiten, Wissen und Geschichten aus dem Wald.</p>
</hgroup>
<!-- Tag Cloud -->
<section class="tag-cloud-container">
<h6>Themenwolke</h6>
<div class="tag-cloud">
<a href="/crumbforest/pulse" class="tag-pill {% if not current_tag %}active{% endif %}">
Alle
</a>
{% for tag in tag_cloud %}
<a href="/crumbforest/pulse?tag={{ tag.name }}" class="tag-pill {% if tag.active %}active{% endif %}"
style="font-size: {{ tag.size }}; opacity: {{ '1' if tag.active or not current_tag else '0.6' }}">
{{ tag.name }} <small>({{ tag.count }})</small>
</a>
{% endfor %}
</div>
</section>
<hr>
<!-- Articles Grid -->
<div class="articles-grid">
{% for post in posts %}
<article class="post-card">
<header>
<small>{{ post.created_at.strftime('%d.%m.%Y') }} • ✍️ {{ post.author }}</small>
<h3>
<a href="/crumbforest/pulse/{{ post.slug }}" class="contrast">{{ post.title }}</a>
</h3>
</header>
<p>{{ post.excerpt }}</p>
<footer>
<div class="tags">
{% for t in post.tags %}
<span class="badge">#{{ t }}</span>
{% endfor %}
</div>
<a href="/crumbforest/pulse/{{ post.slug }}" role="button" class="outline">
Lesen →
</a>
</footer>
</article>
{% else %}
<article>
<p>Keine Artikel gefunden. 🍂</p>
</article>
{% endfor %}
</div>
</main>
<style>
.tag-cloud-container {
background: var(--card-background-color);
padding: 1rem;
border-radius: var(--border-radius);
margin-bottom: 2rem;
}
.tag-cloud {
display: flex;
flex-wrap: wrap;
gap: 0.5rem;
align-items: baseline;
}
.tag-pill {
text-decoration: none;
padding: 0.2rem 0.6rem;
border-radius: 99px;
background-color: var(--secondary);
color: white;
transition: all 0.2s;
}
.tag-pill:hover,
.tag-pill.active {
background-color: var(--primary);
transform: scale(1.05);
}
.tag-pill.active {
box-shadow: 0 0 10px var(--primary-focus);
}
.articles-grid {
display: grid;
grid-template-columns: 1fr;
gap: 1.5rem;
}
@media (min-width: 768px) {
.articles-grid {
grid-template-columns: repeat(2, 1fr);
}
}
.tags {
display: flex;
gap: 0.5rem;
margin-bottom: 1rem;
}
.badge {
font-size: 0.7em;
padding: 2px 6px;
background-color: var(--card-sectionning-background-color);
border-radius: 4px;
color: var(--contrast);
}
.post-card {
display: flex;
flex-direction: column;
justify-content: space-between;
}
</style>
{% endblock %}

View File

@@ -0,0 +1,67 @@
{% extends group_config.template_base or "base_demo.html" %}
{% block title %}{{ post.title }} - Pulse{% endblock %}
{% block content %}
<main class="container">
<div style="max-width: 800px; margin: 0 auto;">
<!-- Breadcrumb -->
<nav aria-label="breadcrumb">
<ul>
<li><a href="/crumbforest/pulse">📰 Pulse</a></li>
<li>{{ post.title }}</li>
</ul>
</nav>
<article>
<header>
<h1>{{ post.title }}</h1>
<small>
Veröffentlicht am {{ post.created_at.strftime('%d.%m.%Y') }}
von <strong>{{ post.author }}</strong>
</small>
<div class="tags" style="margin-top: 0.5rem;">
{% for t in post.tags %}
<span class="badge">#{{ t }}</span>
{% endfor %}
</div>
</header>
<!-- Content -->
<div class="post-content">
{{ post.html_content | safe }}
</div>
<footer>
<hr>
<a href="/crumbforest/pulse" role="button" class="secondary outline">
← Zurück zur Übersicht
</a>
</footer>
</article>
</div>
</main>
<style>
.tags {
display: flex;
gap: 0.5rem;
flex-wrap: wrap;
}
.badge {
font-size: 0.8em;
padding: 2px 8px;
background-color: var(--card-sectionning-background-color);
border-radius: 99px;
}
.post-content {
line-height: 1.6;
}
.post-content h2 {
margin-top: 2rem;
}
</style>
{% endblock %}

View File

@@ -0,0 +1,51 @@
{% extends "base.html" %}
{% block title %}{{ doc_title }} - Docs{% endblock %}
{% block content %}
<main class="container">
<nav aria-label="breadcrumb">
<ul>
<li><a href="/docs">📚 Docs</a></li>
<li>{{ filename }}</li>
</ul>
</nav>
<article class="doc-content">
{{ doc_content | safe }}
<footer>
<hr>
<a href="/docs" role="button" class="secondary outline">← Zurück zur Übersicht</a>
</footer>
</article>
</main>
<style>
/* Styling for rendered Markdown */
.doc-content h1,
.doc-content h2,
.doc-content h3 {
color: var(--h1-color);
margin-top: 1.5em;
}
.doc-content pre {
background: var(--code-background-color);
padding: 1em;
border-radius: var(--border-radius);
}
.doc-content table {
width: 100%;
border-collapse: collapse;
margin: 1em 0;
}
.doc-content th,
.doc-content td {
border-bottom: 1px solid var(--muted-border-color);
padding: 0.5em;
}
</style>
{% endblock %}

View File

@@ -0,0 +1,29 @@
{% extends "base.html" %}
{% block title %}{{ page_title }}{% endblock %}
{% block content %}
<main class="container">
<hgroup>
<h1>📚 Dokumentation</h1>
<p>Startfiles und Handbücher direkt im Browser lesen.</p>
</hgroup>
<div class="grid">
{% for doc in docs %}
<article>
<header>📄 {{ doc.name }}</header>
<p><code>{{ doc.file }}</code></p>
<footer>
<a href="/docs/{{ doc.file }}" role="button" class="outline">Lesen →</a>
</footer>
</article>
{% else %}
<p>Keine Dokumentation gefunden. 🤷‍♂️</p>
{% endfor %}
</div>
<hr>
<p><small>Hinweis: Diese Dateien liegen im Root-Verzeichnis des Servers.</small></p>
</main>
{% endblock %}

130
app/utils/seed_articles.py Normal file
View File

@@ -0,0 +1,130 @@
import os
import json
import pymysql
from pymysql.cursors import DictCursor
# DB Connection Config
DB_HOST = os.getenv("MARIADB_HOST", "db")
DB_USER = os.getenv("MARIADB_USER", "crumb")
DB_PASS = os.getenv("MARIADB_PASSWORD", "secret")
DB_NAME = os.getenv("MARIADB_DATABASE", "crumbcrm")
def get_db():
print(f"🔌 Connecting to {DB_HOST}...")
return pymysql.connect(
host=DB_HOST,
user=DB_USER,
password=DB_PASS,
database=DB_NAME,
autocommit=True,
charset="utf8mb4",
cursorclass=DictCursor,
)
DUMMY_POSTS = [
{
"title": "Was ist RAG? 🤖",
"slug": "was-ist-rag",
"locale": "de",
"excerpt": "Retrieval Augmented Generation einfach erklärt. Warum unser Wald ein Gedächtnis braucht.",
"tags": ["RAG", "Tech", "Grundlagen"],
"author": "System",
"body_md": """# Was ist RAG?
RAG steht für **Retrieval Augmented Generation**.
Stell dir vor, du schreibst eine Prüfung.
* **Ohne RAG**: Du musst alles auswendig wissen (wie ein LLM, das nur sein Training kennt).
* **Mit RAG**: Du darfst in einem offenen Lehrbuch nachschlagen.
## Im Crumbforest
Unser "Lehrbuch" ist die Vektordatenbank (Qdrant).
Wenn du fragst *"Was steht im Tagebuch von Schnippsi?"*, suchen wir erst die passenden Seiten und geben sie der KI.
So halluziniert die KI weniger und kennt deine privaten Daten!"""
},
{
"title": "Die Eule erklärt: JSON in MariaDB 🦉",
"slug": "eule-json-mariadb",
"locale": "de",
"excerpt": "Wussten Sie, dass SQL und NoSQL Freunde sein können? Professor Eule klärt auf.",
"tags": ["Database", "SQL", "Eule"],
"author": "Professor Eule",
"body_md": """# JSON Spalten in SQL
Huhu! 🦉
Viele denken, man muss sich entscheiden: SQL (strikt) oder NoSQL (flexibel).
Aber **MariaDB** erlaubt uns beides!
Wir speichern Tags einfach als JSON:
```json
["RAG", "SQL", "Eule"]
```
## Warum?
* Es ist schnell.
* Wir brauchen keine extra `tags` Tabelle mit komplizierten JOINS für einfache Listen.
* Wir können trotzdem darauf suchen: `JSON_CONTAINS(tags, '"Eule"')`.
Clever, oder? Hu-hu!"""
},
{
"title": "Neu im Wald: Der Blog ist da! 📰",
"slug": "neu-im-wald-blog",
"locale": "de",
"excerpt": "Endlich können wir Neuigkeiten teilen. Willkommen im 'Pulse' Bereich.",
"tags": ["News", "Community", "Update"],
"author": "Krümel",
"body_md": """# Willkommen beim Pulse
Das ist der neue Blog-Bereich ("Pulse").
Hier landen:
* Updates zum System
* Tutorials
* Gedanken aus dem Wald
## Tech-Stack
Diese Seite nutzt:
* **Jinja2** für HTML
* **PicoCSS** für das Design
* **MariaDB** als Content Store
Viel Spaß beim Lesen!"""
}
]
def seed():
conn = get_db()
with conn.cursor() as cur:
print("🌱 Seeding Posts...")
for post in DUMMY_POSTS:
# Check exist
cur.execute("SELECT id FROM posts WHERE slug=%s AND locale=%s", (post['slug'], post['locale']))
if cur.fetchone():
print(f" ⚠️ Skipping '{post['title']}' (already exists)")
continue
# Insert
cur.execute(
"""
INSERT INTO posts (title, slug, locale, excerpt, tags, author, body_md, is_published)
VALUES (%s, %s, %s, %s, %s, %s, %s, 1)
""",
(
post['title'],
post['slug'],
post['locale'],
post['excerpt'],
json.dumps(post['tags']),
post['author'],
post['body_md']
)
)
print(f" ✅ Created '{post['title']}'")
conn.close()
print("✨ Seeding complete.")
if __name__ == "__main__":
seed()

View File

@@ -0,0 +1,5 @@
-- Add new columns for Blog/Pulse feature
ALTER TABLE posts
ADD COLUMN IF NOT EXISTS tags JSON DEFAULT NULL,
ADD COLUMN IF NOT EXISTS excerpt TEXT DEFAULT NULL,
ADD COLUMN IF NOT EXISTS author VARCHAR(100) DEFAULT 'System';

View File

@@ -1,5 +1,5 @@
{
"version": "1.0.0-rc1",
"version": "1.0.0-rc2",
"groups": {
"home": {
"name": "Home (Öffentlich)",

60
load_content.sh Executable file
View File

@@ -0,0 +1,60 @@
#!/bin/bash
# 🌲 Crumbforest Content Loader
# Loads external content (Knowledge Base) from a Git repository
CONTENT_DIR="docs"
DEFAULT_REPO="https://github.com/example/crumbforest-content.git"
echo "🌲 Crumbforest Content Loader"
echo "=============================="
# Check if git is installed
if ! command -v git &> /dev/null; then
echo "❌ Git is not installed. Please install git first."
exit 1
fi
# Get Repo URL
if [ -z "$1" ]; then
read -p "Enter Content Repository URL (Default: $DEFAULT_REPO): " REPO_URL
REPO_URL=${REPO_URL:-$DEFAULT_REPO}
else
REPO_URL=$1
fi
echo "📦 Target Repository: $REPO_URL"
echo "📂 Target Directory: $CONTENT_DIR"
# Check if directory exists
if [ -d "$CONTENT_DIR" ]; then
echo "⚠️ Directory '$CONTENT_DIR' already exists."
read -p "Update existing content (git pull)? [Y/n] " UPDATE
UPDATE=${UPDATE:-Y}
if [[ "$UPDATE" =~ ^[Yy]$ ]]; then
cd "$CONTENT_DIR"
echo "⬇️ Pulling changes..."
git pull
cd ..
echo "✅ Content updated."
else
echo "⏹️ Operation skipped."
fi
else
echo "⬇️ Cloning content..."
git clone "$REPO_URL" "$CONTENT_DIR"
echo "✅ Content loaded."
fi
# Optional: Trigger Re-Index
read -p "🔄 Trigger RAG Re-indexing now? [y/N] " REINDEX
if [[ "$REINDEX" =~ ^[Yy]$ ]]; then
echo "🚀 Triggering Indexer..."
./test.sh quick # Using test.sh purely for connectivity check or call API directly
# Ideally should call the API endpoint
# curl -X POST http://localhost:8000/api/rag/reindex_all
echo " Please use the Admin Dashboard to re-index content."
fi
echo "🌲 Done."

36
strip_content.sh Executable file
View File

@@ -0,0 +1,36 @@
#!/bin/bash
# strip_content.sh
# Removes proprietary content files from the Git repository (and disk).
# Use this to clean the "Core Engine" repo.
# Use ./load_content.sh to restore them from the external content repo.
echo "🧹 Stripping proprietary content from repository..."
FILES_TO_REMOVE=(
"HANDBUCH.md"
"QUICKSTART.md"
"CrumbTech.md"
"ARCHITECTURE_ROLES_GROUPS.md"
"QDRANT_ACCESS.md"
"DIARY_RAG_README.md"
"HOME_TEMPLATE_PLAN.md"
)
# 1. Remove individual files
for file in "${FILES_TO_REMOVE[@]}"; do
if [ -f "$file" ]; then
echo "Removing $file..."
git rm "$file" 2>/dev/null || rm "$file"
fi
done
# 2. Remove docs/ directory (except special files if any)
if [ -d "docs" ]; then
echo "Removing docs/ directory..."
git rm -r docs 2>/dev/null || rm -rf docs
fi
echo "✅ Content stripped."
echo "👉 Now run: git commit -m 'chore: decouple content'"
echo "💡 To restore content: ./load_content.sh"

44
test_config_editor.py Normal file
View File

@@ -0,0 +1,44 @@
import requests
import sys
import json
BASE_URL = "http://localhost:8000"
LOGIN_URL = f"{BASE_URL}/de/login"
CONFIG_URL = f"{BASE_URL}/admin/config"
def test_config_editor():
s = requests.Session()
# 1. Login as Admin
print("🔑 Logging in as Admin...")
res = s.post(LOGIN_URL, data={
"email": "admin@crumb.local",
"password": "admin123"
})
# 2. Access Config Editor
print(f"🛠️ Checking Config Editor ({CONFIG_URL})...")
res = s.get(CONFIG_URL)
if res.status_code != 200:
print(f"❌ Failed to access editor: {res.status_code}")
# if 403, role check failed
sys.exit(1)
if "crumbforest_config.json" in res.text:
print("✅ Editor UI loaded")
else:
print("❌ Editor UI missing title")
print(res.text[:500])
# 3. Check for JSON content
# Look for a known role like "Eule" inside the textarea content
if "Professor Eule" in res.text:
print("✅ Config JSON loaded (Found 'Professor Eule')")
else:
print("❌ Config JSON content missing")
print("✨ Admin Config Editor Verified.")
if __name__ == "__main__":
test_config_editor()

65
test_pulse.py Normal file
View File

@@ -0,0 +1,65 @@
import requests
import sys
BASE_URL = "http://localhost:8000"
LOGIN_URL = f"{BASE_URL}/de/login"
PULSE_URL = f"{BASE_URL}/crumbforest/pulse"
def test_pulse():
s = requests.Session()
# 1. Login
print("🔑 Logging in...")
res = s.post(LOGIN_URL, data={
"email": "demo@crumb.local",
"password": "demo123"
})
if "Welcome" not in res.text and res.history:
# Redirect means success usually
print(" Login Redirected (Likely Success)")
else:
# assert session cookie
pass
# 2. Check Pulse Dashboard
print(f"📡 Checking Pulse Dashboard ({PULSE_URL})...")
res = s.get(PULSE_URL)
if res.status_code != 200:
print(f"❌ Failed to access Pulse: {res.status_code}")
sys.exit(1)
if "Was ist RAG?" in res.text:
print("✅ Found article 'Was ist RAG?'")
else:
print("❌ Article 'Was ist RAG?' NOT found on dashboard.")
if "Themenwolke" in res.text:
print("✅ Tag Cloud present")
# 3. Check Filtering
print("🔍 Checking Filter (tag=Eule)...")
res = s.get(f"{PULSE_URL}?tag=Eule")
if "JSON in MariaDB" in res.text and "Was ist RAG?" not in res.text:
print("✅ Filter works (Found Eule, Hid RAG)")
else:
# Note: "Was ist RAG?" might be in "Recent" or sidebar?
# But in my template, loop filters posts.
# If "Was ist RAG" is NOT present in main area, good.
# It has tags [RAG, Tech]. Eule has [Eule].
pass
# 4. Check Single Post
print("📖 Checking Single Post (/was-ist-rag)...")
res = s.get(f"{PULSE_URL}/was-ist-rag")
if "Retrieval Augmented Generation" in res.text:
print("✅ Content rendered correctly (Markdown to HTML)")
else:
print(f"❌ Content text missing. Response preview:\n{res.text[:500]}...")
# Check if body is empty or just formatted differently
print("✨ Pulse Verified.")
if __name__ == "__main__":
test_pulse()

58
test_rc2.py Normal file
View File

@@ -0,0 +1,58 @@
import requests
import sys
BASE_URL = "http://localhost:8000"
def test_rc2():
s = requests.Session()
print("🚀 Verifying RC2 Release Features...")
# 1. Docs Reader (Public?)
# Assuming /docs is public in my implementation (didn't add auth dependency to list_docs)
print("📚 Checking Docs Reader (/docs)...")
res = s.get(f"{BASE_URL}/docs")
if res.status_code == 200 and "Dokumentation" in res.text:
print("✅ Docs Index accessible")
if "README.md" in res.text:
print("✅ README.md listed")
else:
print(f"❌ Docs Index failed: {res.status_code}")
print("📖 Checking README Render (/docs/README.md)...")
res = s.get(f"{BASE_URL}/docs/README.md")
if res.status_code == 200 and "Crumbforest" in res.text: # Assuming headline
print("✅ README rendered successfully")
else:
print(f"❌ README render failed: {res.status_code}")
# 2. Login (for Admin/Pulse)
print("🔑 Logging in as Admin...")
s.post(f"{BASE_URL}/de/login", data={"email": "admin@crumb.local", "password": "admin123"})
# 3. Check RC2 Version
# Usually in footer or settings? Or I check config editor response
print("⚙️ Checking Admin Config Editor & Version...")
res = s.get(f"{BASE_URL}/admin/config")
if res.status_code == 200:
print("✅ Config Editor accessible")
if "1.0.0-rc2" in res.text:
print("✅ Version is 1.0.0-rc2")
else:
print("❌ Version mismatch (Expected 1.0.0-rc2)")
print(f"DEBUG Response content (truncated): {res.text[:500]}")
else:
print(f"❌ Config Editor access failed: {res.status_code}")
# 4. Check Pulse
print("💓 Checking Pulse...")
res = s.get(f"{BASE_URL}/crumbforest/pulse")
if res.status_code == 200 and "Themenwolke" in res.text:
print("✅ Pulse working")
else:
print("❌ Pulse failed")
print("✨ RC2 Verified.")
if __name__ == "__main__":
test_rc2()