Compare commits

...

11 Commits

Author SHA1 Message Date
DXC
c6b9783a7a fix(janitor): 禁止启动时因目录不存在而自动删除数据库任务记录
reconcile_tasks_on_startup() 中有一段逻辑:遍历所有任务,如果 output_dir
在磁盘上不存在,直接 DELETE FROM tasks。这导致重启时如果目录被移动/重命名/
被之前的 janitor 清理,所有任务记录静默丢失。

修改为仅输出 WARNING 日志,保留数据库记录。用户可通过 DELETE /task/<id> 手动清理。
2026-07-13 10:40:31 +08:00
DXC
55cb7fb025 fix(gasflux.ini): 禁止失败任务的自动清理
failed_task_cleanup_age: 60 → 315360000(10年)
janitor 每30秒扫描 + 启动时 reconcile 都会清理超过此时间的失败任务,
60秒太短,改为10年等同于禁用自动清理
2026-07-10 17:04:51 +08:00
DXC
adaabb2831 fix(plotting): 风速时序图增加 winddir 通道
_time_series_wrapper 的 ys 参数从 ['windspeed'] 改为 ['windspeed', 'winddir'],
双Y轴分别显示风速 (m/s) 和风向 (°)
2026-07-10 16:38:48 +08:00
DXC
9bafc9ba0e revert(plotting): 恢复原始 plotting.py 并添加适配层
1. 从初始提交 (f085d7c) 恢复完整的原始 plotting.py
   - 原始配色: geyser 色阶, simple_white 模板
   - 风玫瑰: Beaufort 12级分桶 + turquoise 扇形
   - 时序图: 散点 + 滚动均线 + 多Y轴
   - 背景校正: 双Y轴 + timestamp + 虚线基线
   - 3D散点: px.scatter_3d + 自定义 hover
   - 克里金: 散点叠加等高线 + 地面线

2. 新增适配包装函数(原始签名与当前调用方式不兼容):
   - _scatter_3d_wrapper(df,gas) → scatter_3d(df,color=...)
   - _windrose_wrapper(df) → windrose(df)
   - _time_series_wrapper(df) → time_series(df,ys=['windspeed'])
   - _contour_krig_wrapper(krig_vars) → contour_krig(df,gas,xx,yy,field)
   - _heatmap_krig_wrapper(krig_vars) → heatmap_krig(xx,yy,field)
   - _semivariogram_plot(variogram,gas) → 保留 Plotly 实现
   - _outliers_wrapper(df,col,name) → outliers(series,high,low)

3. 更新调用方使用包装函数:
   - processing_pipelines.py: _scatter_3d_wrapper/_windrose_wrapper/_time_series_wrapper
   - interpolation.py: _contour_krig_wrapper/_heatmap_krig_wrapper/_semivariogram_plot
   - pre_processing.py: _outliers_wrapper
2026-07-10 16:34:49 +08:00
DXC
9898b68410 refactor(download): 移除下载后自动删除逻辑,保留手动删除接口
1. download.py:
   - 完全移除 _mark_task_downloaded() 函数(原功能:下载后设置 delete_after_at
     触发 janitor 到期自动清理文件)
   - 改为仅记录 downloaded_at 时间戳,不再调度自动删除
   - 手动 DELETE /task/<id> 接口不受影响(tasks.py:151-211)

2. gasflux.ini:
   - successful_task_cleanup_age: 60 → 315360000(10年)
     彻底禁止 janitor 的下载后自动清理行为
2026-07-10 16:20:49 +08:00
DXC
fa0cd769a8 fix(build_exe.bat): 修复 PyInstaller 打包脚本——补充缺失依赖和调整配置
1. 补充 25+ 个 hidden-import(打包后运行崩溃的根因):
   plotly/pybaselines/scipy子模块/molmass/openpyxl/pyproj/jinja2/
   requests/urllib3/certifi/geopandas/shapely/fiona/simplekml/joblib/
   skgstat.DirectionalVariogram/tqdm 等

2. 补充 --add-data 文件(运行时 FileNotFound 的根因):
   mass_balance_template.html / gasflux.ini / gasflux.ini.example

3. 修复 python 命令兼容性:改为 py -3(适配 Windows Store Python)

4. 修正端口号提示:5000 → 5001(与 gasflux.ini 一致)
2026-07-10 15:20:16 +08:00
DXC
3367774574 fix(server_waitress.py): 修复 PyInstaller 打包后 UTF-8 编码包装导致的启动崩溃
sys.stdout/sys.stderr 包装为 UTF-8 TextIOWrapper 时,
在 PyInstaller --onefile 无控制台模式或管道重定向场景下,
sys.stdout.buffer 可能不存在,抛出 AttributeError 导致 EXE 启动失败。
增加 try/except 捕获 AttributeError 和 ValueError,失败时跳过编码包装。
2026-07-10 15:20:03 +08:00
DXC
616783fb52 feat(reporting.py): 将 Plotly 图表转换为内嵌 HTML 并嵌入报告
mass_balance_report() 函数:
1. 图表→HTML 转换:
   - 原代码: 所有 plot_htmls 值为空字符串(图表禁用)
   - 修复: 新增 _fig_to_html() 辅助函数,通过 fig.to_html(full_html=False,
     include_plotlyjs=True) 将 5 个 Plotly Figure 对象分别转换为
     HTML div+script 片段,内嵌完整 Plotly.js 库(~3MB)
     对应映射: threed_fig→"3D", krig_fig→"krig",
              windrose_fig→"windrose", wind_fig→"wind",
              background_fig→"background"

2. CDN→内嵌: include_plotlyjs='cdn' 改为 include_plotlyjs=True,
   解决了离线打开 HTML 报告时 'Plotly is not defined' 的 JavaScript 错误,
   以及 file:// 协议下无法加载外部 CDN 资源的安全限制

影响范围:HTML 报告大小从 ~170KB 增至 ~15MB,支持完全离线查看,5 张交互式图表均可正常渲染
2026-07-10 14:49:30 +08:00
DXC
4ec46208dd feat(processing_pipelines.py): 启用 3D散点/风玫瑰/风速时序图表创建
两处修改:

1. InSituSensorStrategy.process():
   - 原代码: scatter_3d[gas]=None, windrose=None, wind_timeseries=None
   - 修复: 调用 plotting.scatter_3d()/windrose()/time_series() 创建实际图表,
     每个调用独立 try/except 保护,失败时回退至 blank_figure()

2. 模块导入:
   - 新增 import plotly.graph_objects as go
     供 Curtain/Spiral 空间策略中的 fig.add_trace(go.Scatter3d(...)) 使用
     (该代码此前存在但从未执行,因为 fig 始终为 None)

影响范围:HTML 报告中 3D 飞行轨迹图、风玫瑰图、风速时序图从空白变为正常显示
2026-07-10 14:49:22 +08:00
DXC
5609a8d708 feat(pre_processing.py): 启用异常值检测图表生成
remove_outliers() 函数:
- 原代码: fig = None  # plotting disabled
- 修复: 调用 plotting.outliers(df, column, name) 生成箱线图,
  显示 IQR 方法的异常值检测结果

异常保护:try/except 包裹,绘图失败时回退至 None
2026-07-10 14:49:14 +08:00
DXC
e5f43f0297 fix(interpolation.py): 修复 krig_variables 变量先使用后定义的 NameError
ordinary_kriging() 函数存在两个问题:

1. krig_variables 先使用后定义(NameError 被 try/except 静默吞掉):
   - 原代码: 第103-114行绘图调用引用了 krig_variables,
     但该字典在第121-133行才创建
   - 修复: 将 krig_variables 字典构造移至绘图调用之前(第103行之前),
     确保 contour_krig()、heatmap_krig()、semivariogram_plot() 可正常获取数据

2. 启用克里金图表输出:
   - 原代码: contour_plot = None; grid_plot = None; semivariogram_plot = None
   - 修复: 分别调用 plotting.contour_krig()、plotting.heatmap_krig()、
     plotting.semivariogram_plot(),每个调用独立 try/except 保护

影响范围:克里金等高线图和热力图从永远空白变为正常显示
2026-07-10 14:49:09 +08:00
10 changed files with 728 additions and 437 deletions

View File

@ -5,19 +5,19 @@ REM This script builds a standalone executable using Waitress WSGI server
echo Building GasFlux Web API executable... echo Building GasFlux Web API executable...
REM Check if PyInstaller is installed REM Check if PyInstaller is installed
python -c "import PyInstaller" >nul 2>&1 py -3 -c "import PyInstaller" >nul 2>&1
if errorlevel 1 ( if errorlevel 1 (
echo Error: PyInstaller is not installed. Please run: echo Error: PyInstaller is not installed. Please run:
echo pip install pyinstaller waitress echo py -3 -m pip install pyinstaller waitress
pause pause
exit /b 1 exit /b 1
) )
REM Check if Waitress is installed REM Check if Waitress is installed
python -c "import waitress" >nul 2>&1 py -3 -c "import waitress" >nul 2>&1
if errorlevel 1 ( if errorlevel 1 (
echo Error: Waitress is not installed. Please run: echo Error: Waitress is not installed. Please run:
echo pip install waitress echo py -3 -m pip install waitress
pause pause
exit /b 1 exit /b 1
) )
@ -41,8 +41,24 @@ pyinstaller --onefile ^
--hidden-import skgstat ^ --hidden-import skgstat ^
--hidden-import skgstat.Variogram ^ --hidden-import skgstat.Variogram ^
--hidden-import skgstat.OrdinaryKriging ^ --hidden-import skgstat.OrdinaryKriging ^
--add-data "src\gasflux\gasflux_config.yaml;src\gasflux" ^ --hidden-import skgstat.DirectionalVariogram ^
--add-data "API_DOCUMENTATION.md;." ^ --hidden-import plotly ^
--hidden-import plotly.graph_objects ^
--hidden-import plotly.express ^
--hidden-import plotly.subplots ^
--hidden-import pybaselines ^
--hidden-import scipy ^
--hidden-import scipy.odr ^
--hidden-import scipy.signal ^
--hidden-import scipy.stats ^
--hidden-import scipy.integrate ^
--hidden-import scipy.optimize ^
--hidden-import molmass ^
--hidden-import openpyxl ^
--hidden-import pyproj ^
--hidden-import jinja2 ^
--hidden-import requests ^
--hidden-import urllib3 ^
--hidden-import matplotlib ^ --hidden-import matplotlib ^
--hidden-import matplotlib.pyplot ^ --hidden-import matplotlib.pyplot ^
--hidden-import matplotlib.backends ^ --hidden-import matplotlib.backends ^
@ -53,7 +69,19 @@ pyinstaller --onefile ^
--hidden-import matplotlib.patches ^ --hidden-import matplotlib.patches ^
--hidden-import matplotlib.text ^ --hidden-import matplotlib.text ^
--hidden-import matplotlib.transforms ^ --hidden-import matplotlib.transforms ^
--hidden-import tqdm ^
--hidden-import certifi ^
--hidden-import charset_normalizer ^
--hidden-import geopandas ^
--hidden-import shapely ^
--hidden-import fiona ^
--hidden-import simplekml ^
--hidden-import joblib ^
--exclude-module tkinter ^ --exclude-module tkinter ^
--add-data "src\gasflux\gasflux_config.yaml;src\gasflux" ^
--add-data "src\gasflux\templates\mass_balance_template.html;src\gasflux\templates" ^
--add-data "gasflux.ini;." ^
--add-data "gasflux.ini.example;." ^
server_waitress.py server_waitress.py
if errorlevel 1 ( if errorlevel 1 (
@ -69,6 +97,8 @@ echo.
echo To run the server: echo To run the server:
echo GasFluxAPI.exe echo GasFluxAPI.exe
echo. echo.
echo The server will start on http://localhost:5000 echo The server will start on http://localhost:5001
echo Make sure gasflux.ini is in the same directory as the executable,
echo or configure paths via environment variables.
echo. echo.
pause pause

View File

@ -30,9 +30,9 @@ task_cleanup_interval = 30
# 24 hours in seconds # 24 hours in seconds
max_task_age = 86400 max_task_age = 86400
# 1 minute in seconds for successful tasks # 1 minute in seconds for successful tasks
successful_task_cleanup_age = 60 successful_task_cleanup_age = 315360000
# 1 minute in seconds for failed tasks # 1 minute in seconds for failed tasks
failed_task_cleanup_age = 60 failed_task_cleanup_age = 315360000
janitor_dry_run = false janitor_dry_run = false
[performance] [performance]

View File

@ -45,9 +45,17 @@ import sys
from pathlib import Path from pathlib import Path
# 设置控制台编码为 UTF-8(解决 Windows 中文乱码问题) # 设置控制台编码为 UTF-8(解决 Windows 中文乱码问题)
# PyInstaller onefile 模式下 sys.stdout.buffer 可能不存在(无控制台或管道重定向),
# 此时跳过编码包装,避免 AttributeError 导致启动崩溃
import io import io
try:
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8') sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')
except (AttributeError, ValueError):
pass
try:
sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8') sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8')
except (AttributeError, ValueError):
pass
# Add the project root and src directory to PYTHONPATH # Add the project root and src directory to PYTHONPATH
project_root = Path(__file__).parent.absolute() project_root = Path(__file__).parent.absolute()

View File

@ -3,6 +3,7 @@ Download Blueprint
Handles file download endpoints. Handles file download endpoints.
""" """
from datetime import datetime
from pathlib import Path from pathlib import Path
from flask import Blueprint, send_file, current_app from flask import Blueprint, send_file, current_app
@ -10,40 +11,6 @@ from ..shared import _format_response, log_performance, logger
from ..auth import require_api_key from ..auth import require_api_key
def _mark_task_downloaded(task_id):
"""Mark task as downloaded and schedule deletion in database."""
import sqlite3
from pathlib import Path
# Use independent database connection (not from flask.g which may be closed)
from ..db import get_db_path as get_config_db_path
db_path = get_config_db_path(current_app)
# Get cleanup age for successful tasks from config (in seconds)
successful_task_cleanup_age = current_app.config.get('SUCCESSFUL_TASK_CLEANUP_AGE', 3600)
try:
conn = sqlite3.connect(str(db_path), check_same_thread=False)
conn.execute("PRAGMA foreign_keys=ON")
conn.execute("PRAGMA busy_timeout=3000")
# Update downloaded timestamp and set deletion time based on config
conn.execute("""
UPDATE tasks
SET downloaded_at = datetime('now', '+8 hours'),
delete_after_at = datetime('now', '+8 hours', '+' || ? || ' seconds')
WHERE task_id = ?
""", (successful_task_cleanup_age, task_id))
conn.commit()
logger.info(f"Task {task_id} marked as downloaded, scheduled for deletion in {successful_task_cleanup_age} seconds")
except Exception as e:
logger.error(f"Failed to mark task {task_id} as downloaded: {str(e)}", exc_info=True)
finally:
if 'conn' in locals():
conn.close()
# Create blueprint # Create blueprint
download_bp = Blueprint('download', __name__, url_prefix='/download') download_bp = Blueprint('download', __name__, url_prefix='/download')
@ -124,12 +91,19 @@ def download_file(filename):
file_size = file_path.stat().st_size file_size = file_path.stat().st_size
logger.info(f"Serving file: {filename} ({file_size} bytes)") logger.info(f"Serving file: {filename} ({file_size} bytes)")
# Mark download immediately before sending file # 记录下载时间(仅时间戳,不设置自动删除)
if task_id: if task_id:
try: try:
_mark_task_downloaded(task_id) from ..db import get_db
db = get_db()
db.execute(
"UPDATE tasks SET downloaded_at = datetime('now', '+8 hours') WHERE task_id = ?",
(task_id,)
)
db.commit()
logger.info(f"Task {task_id} downloaded at {datetime.now()}")
except Exception as e: except Exception as e:
logger.error(f"Failed to mark download for task {task_id}: {str(e)}") logger.error(f"Failed to record download for task {task_id}: {str(e)}")
response = send_file(file_path) response = send_file(file_path)
return response return response

View File

@ -99,11 +99,6 @@ def ordinary_kriging(
# np.nan_to_num(error_1s, copy=False, nan=0) # np.nan_to_num(error_1s, copy=False, nan=0)
volume_error = simpsonintegrate(error_1s, x_cell_size, y_cell_size) volume_error = simpsonintegrate(error_1s, x_cell_size, y_cell_size)
# Plots disabled
contour_plot = None
grid_plot = None
semivariogram_plot = None
output_text = ( output_text = (
f"The emissions flux of {gas.upper()} is {volume:.3f}kgh⁻¹; " f"The emissions flux of {gas.upper()} is {volume:.3f}kgh⁻¹; "
f"the cut and fill volumes of the grid are {volumepos:.3f} and {volumeneg:.3f}kgh⁻¹. " f"the cut and fill volumes of the grid are {volumepos:.3f} and {volumeneg:.3f}kgh⁻¹. "
@ -123,6 +118,20 @@ def ordinary_kriging(
"volume_error": volume_error, "volume_error": volume_error,
} }
# Generate plots (must be after krig_variables is defined)
try:
contour_plot = plotting._contour_krig_wrapper(krig_variables)
except Exception:
contour_plot = None
try:
grid_plot = plotting._heatmap_krig_wrapper(krig_variables)
except Exception:
grid_plot = None
try:
semivariogram_plot = plotting._semivariogram_plot(semivariogram, gas)
except Exception:
semivariogram_plot = None
return krig_variables, output_text, contour_plot, grid_plot, semivariogram_plot return krig_variables, output_text, contour_plot, grid_plot, semivariogram_plot

View File

@ -310,8 +310,7 @@ def reconcile_tasks_on_startup():
except Exception as e: except Exception as e:
current_app.logger.error(f"Failed to delete failed task {task_id}: {str(e)}", exc_info=True) current_app.logger.error(f"Failed to delete failed task {task_id}: {str(e)}", exc_info=True)
# Check for tasks with output directories that no longer exist # 检查 output_dir 不存在的任务(仅警告,不自动删除)
# This helps clean up database entries for manually deleted directories
rows = conn.execute(""" rows = conn.execute("""
SELECT task_id, output_dir SELECT task_id, output_dir
FROM tasks FROM tasks
@ -321,18 +320,21 @@ def reconcile_tasks_on_startup():
orphaned_count = 0 orphaned_count = 0
for row in rows: for row in rows:
task_id_from_db = row[0] # task_id是第一个字段 task_id_from_db = row[0]
output_dir_from_db = row[1] # output_dir是第二个字段 output_dir_from_db = row[1]
if not Path(output_dir_from_db).exists(): if not Path(output_dir_from_db).exists():
conn.execute( current_app.logger.warning(
"DELETE FROM tasks WHERE task_id = ?", f"Startup reconciliation: Task {task_id_from_db} output directory "
(task_id_from_db,) f"not found on disk: {output_dir_from_db}. Database record preserved."
) )
orphaned_count += 1 orphaned_count += 1
if orphaned_count > 0: if orphaned_count > 0:
current_app.logger.info(f"Startup reconciliation: Marked {orphaned_count} tasks as deleted (directories not found)") current_app.logger.info(
f"Startup reconciliation: Found {orphaned_count} tasks with missing directories "
f"(records preserved, use DELETE /task/<id> to manually clean up)"
)
# Backfill output_dir for rows with NULL, using OUTPUT_FOLDER/<task_id> # Backfill output_dir for rows with NULL, using OUTPUT_FOLDER/<task_id>
output_base = Path(current_app.config.get('OUTPUT_FOLDER') or '') output_base = Path(current_app.config.get('OUTPUT_FOLDER') or '')

File diff suppressed because it is too large Load Diff

View File

@ -129,7 +129,11 @@ def remove_outliers(df: pd.DataFrame, column: str, name: str):
fence_low = q1 - 3 * iqr fence_low = q1 - 3 * iqr
fence_high = q3 + 3 * iqr fence_high = q3 + 3 * iqr
fig = None # plotting disabled try:
from . import plotting
fig = plotting._outliers_wrapper(df, column, name)
except Exception:
fig = None
outliers = df.loc[(df[column] < fence_low) | (df[column] > fence_high)] outliers = df.loc[(df[column] < fence_low) | (df[column] > fence_high)]
if len(outliers) > 0: if len(outliers) > 0:

View File

@ -5,6 +5,7 @@ from scipy import stats
import pandas as pd import pandas as pd
import yaml import yaml
import plotly.graph_objects as go
from src.gasflux import background,plotting,processing,reporting,interpolation,pre_processing,gas from src.gasflux import background,plotting,processing,reporting,interpolation,pre_processing,gas
@ -123,9 +124,22 @@ class InSituSensorStrategy(SensorStrategy):
def process(self): def process(self):
logger.info("Processing in-situ (point) data") logger.info("Processing in-situ (point) data")
for gas in self.data_processor.gases: for gas in self.data_processor.gases:
self.data_processor.figs["scatter_3d"][gas] = None # Create 3D scatter plot of flight path with gas concentration
self.data_processor.figs["windrose"] = None try:
self.data_processor.figs["wind_timeseries"] = None self.data_processor.figs["scatter_3d"][gas] = plotting._scatter_3d_wrapper(
self.data_processor.df, gas
)
except Exception:
self.data_processor.figs["scatter_3d"][gas] = plotting.blank_figure()
# Create wind rose and time series plots
try:
self.data_processor.figs["windrose"] = plotting._windrose_wrapper(self.data_processor.df)
except Exception:
self.data_processor.figs["windrose"] = plotting.blank_figure()
try:
self.data_processor.figs["wind_timeseries"] = plotting._time_series_wrapper(self.data_processor.df)
except Exception:
self.data_processor.figs["wind_timeseries"] = plotting.blank_figure()
class SpatialProcessingStrategy(ABC): class SpatialProcessingStrategy(ABC):

View File

@ -27,13 +27,22 @@ def mass_balance_report(
"""Generate a mass balance report (plots disabled).""" """Generate a mass balance report (plots disabled)."""
template_path = Path(__file__).parent / "templates" / "mass_balance_template.html" template_path = Path(__file__).parent / "templates" / "mass_balance_template.html"
# Plots disabled -> use empty strings # Convert figures to HTML strings
def _fig_to_html(fig) -> str:
"""Safely convert a plotly figure to HTML string."""
if fig is None:
return ""
try:
return fig.to_html(full_html=False, include_plotlyjs=True)
except Exception:
return ""
plot_htmls = { plot_htmls = {
"3D": "", "3D": _fig_to_html(threed_fig),
"krig": "", "krig": _fig_to_html(krig_fig),
"windrose": "", "windrose": _fig_to_html(windrose_fig),
"wind": "", "wind": _fig_to_html(wind_fig),
"background": "", "background": _fig_to_html(background_fig),
} }
summary_data = { summary_data = {