Compare commits

...

10 Commits

Author SHA1 Message Date
DXC
55cb7fb025 fix(gasflux.ini): 禁止失败任务的自动清理
failed_task_cleanup_age: 60 → 315360000(10年)
janitor 每30秒扫描 + 启动时 reconcile 都会清理超过此时间的失败任务,
60秒太短,改为10年等同于禁用自动清理
2026-07-10 17:04:51 +08:00
DXC
adaabb2831 fix(plotting): 风速时序图增加 winddir 通道
_time_series_wrapper 的 ys 参数从 ['windspeed'] 改为 ['windspeed', 'winddir'],
双Y轴分别显示风速 (m/s) 和风向 (°)
2026-07-10 16:38:48 +08:00
DXC
9bafc9ba0e revert(plotting): 恢复原始 plotting.py 并添加适配层
1. 从初始提交 (f085d7c) 恢复完整的原始 plotting.py
   - 原始配色: geyser 色阶, simple_white 模板
   - 风玫瑰: Beaufort 12级分桶 + turquoise 扇形
   - 时序图: 散点 + 滚动均线 + 多Y轴
   - 背景校正: 双Y轴 + timestamp + 虚线基线
   - 3D散点: px.scatter_3d + 自定义 hover
   - 克里金: 散点叠加等高线 + 地面线

2. 新增适配包装函数(原始签名与当前调用方式不兼容):
   - _scatter_3d_wrapper(df,gas) → scatter_3d(df,color=...)
   - _windrose_wrapper(df) → windrose(df)
   - _time_series_wrapper(df) → time_series(df,ys=['windspeed'])
   - _contour_krig_wrapper(krig_vars) → contour_krig(df,gas,xx,yy,field)
   - _heatmap_krig_wrapper(krig_vars) → heatmap_krig(xx,yy,field)
   - _semivariogram_plot(variogram,gas) → 保留 Plotly 实现
   - _outliers_wrapper(df,col,name) → outliers(series,high,low)

3. 更新调用方使用包装函数:
   - processing_pipelines.py: _scatter_3d_wrapper/_windrose_wrapper/_time_series_wrapper
   - interpolation.py: _contour_krig_wrapper/_heatmap_krig_wrapper/_semivariogram_plot
   - pre_processing.py: _outliers_wrapper
2026-07-10 16:34:49 +08:00
DXC
9898b68410 refactor(download): 移除下载后自动删除逻辑,保留手动删除接口
1. download.py:
   - 完全移除 _mark_task_downloaded() 函数(原功能:下载后设置 delete_after_at
     触发 janitor 到期自动清理文件)
   - 改为仅记录 downloaded_at 时间戳,不再调度自动删除
   - 手动 DELETE /task/<id> 接口不受影响(tasks.py:151-211)

2. gasflux.ini:
   - successful_task_cleanup_age: 60 → 315360000(10年)
     彻底禁止 janitor 的下载后自动清理行为
2026-07-10 16:20:49 +08:00
DXC
fa0cd769a8 fix(build_exe.bat): 修复 PyInstaller 打包脚本——补充缺失依赖和调整配置
1. 补充 25+ 个 hidden-import(打包后运行崩溃的根因):
   plotly/pybaselines/scipy子模块/molmass/openpyxl/pyproj/jinja2/
   requests/urllib3/certifi/geopandas/shapely/fiona/simplekml/joblib/
   skgstat.DirectionalVariogram/tqdm 等

2. 补充 --add-data 文件(运行时 FileNotFound 的根因):
   mass_balance_template.html / gasflux.ini / gasflux.ini.example

3. 修复 python 命令兼容性:改为 py -3(适配 Windows Store Python)

4. 修正端口号提示:5000 → 5001(与 gasflux.ini 一致)
2026-07-10 15:20:16 +08:00
DXC
3367774574 fix(server_waitress.py): 修复 PyInstaller 打包后 UTF-8 编码包装导致的启动崩溃
sys.stdout/sys.stderr 包装为 UTF-8 TextIOWrapper 时,
在 PyInstaller --onefile 无控制台模式或管道重定向场景下,
sys.stdout.buffer 可能不存在,抛出 AttributeError 导致 EXE 启动失败。
增加 try/except 捕获 AttributeError 和 ValueError,失败时跳过编码包装。
2026-07-10 15:20:03 +08:00
DXC
616783fb52 feat(reporting.py): 将 Plotly 图表转换为内嵌 HTML 并嵌入报告
mass_balance_report() 函数:
1. 图表→HTML 转换:
   - 原代码: 所有 plot_htmls 值为空字符串(图表禁用)
   - 修复: 新增 _fig_to_html() 辅助函数,通过 fig.to_html(full_html=False,
     include_plotlyjs=True) 将 5 个 Plotly Figure 对象分别转换为
     HTML div+script 片段,内嵌完整 Plotly.js 库(~3MB)
     对应映射: threed_fig→"3D", krig_fig→"krig",
              windrose_fig→"windrose", wind_fig→"wind",
              background_fig→"background"

2. CDN→内嵌: include_plotlyjs='cdn' 改为 include_plotlyjs=True,
   解决了离线打开 HTML 报告时 'Plotly is not defined' 的 JavaScript 错误,
   以及 file:// 协议下无法加载外部 CDN 资源的安全限制

影响范围:HTML 报告大小从 ~170KB 增至 ~15MB,支持完全离线查看,5 张交互式图表均可正常渲染
2026-07-10 14:49:30 +08:00
DXC
4ec46208dd feat(processing_pipelines.py): 启用 3D散点/风玫瑰/风速时序图表创建
两处修改:

1. InSituSensorStrategy.process():
   - 原代码: scatter_3d[gas]=None, windrose=None, wind_timeseries=None
   - 修复: 调用 plotting.scatter_3d()/windrose()/time_series() 创建实际图表,
     每个调用独立 try/except 保护,失败时回退至 blank_figure()

2. 模块导入:
   - 新增 import plotly.graph_objects as go
     供 Curtain/Spiral 空间策略中的 fig.add_trace(go.Scatter3d(...)) 使用
     (该代码此前存在但从未执行,因为 fig 始终为 None)

影响范围:HTML 报告中 3D 飞行轨迹图、风玫瑰图、风速时序图从空白变为正常显示
2026-07-10 14:49:22 +08:00
DXC
5609a8d708 feat(pre_processing.py): 启用异常值检测图表生成
remove_outliers() 函数:
- 原代码: fig = None  # plotting disabled
- 修复: 调用 plotting.outliers(df, column, name) 生成箱线图,
  显示 IQR 方法的异常值检测结果

异常保护:try/except 包裹,绘图失败时回退至 None
2026-07-10 14:49:14 +08:00
DXC
e5f43f0297 fix(interpolation.py): 修复 krig_variables 变量先使用后定义的 NameError
ordinary_kriging() 函数存在两个问题:

1. krig_variables 先使用后定义(NameError 被 try/except 静默吞掉):
   - 原代码: 第103-114行绘图调用引用了 krig_variables,
     但该字典在第121-133行才创建
   - 修复: 将 krig_variables 字典构造移至绘图调用之前(第103行之前),
     确保 contour_krig()、heatmap_krig()、semivariogram_plot() 可正常获取数据

2. 启用克里金图表输出:
   - 原代码: contour_plot = None; grid_plot = None; semivariogram_plot = None
   - 修复: 分别调用 plotting.contour_krig()、plotting.heatmap_krig()、
     plotting.semivariogram_plot(),每个调用独立 try/except 保护

影响范围:克里金等高线图和热力图从永远空白变为正常显示
2026-07-10 14:49:09 +08:00
9 changed files with 718 additions and 429 deletions

View File

@ -5,19 +5,19 @@ REM This script builds a standalone executable using Waitress WSGI server
echo Building GasFlux Web API executable...
REM Check if PyInstaller is installed
python -c "import PyInstaller" >nul 2>&1
py -3 -c "import PyInstaller" >nul 2>&1
if errorlevel 1 (
echo Error: PyInstaller is not installed. Please run:
echo pip install pyinstaller waitress
echo py -3 -m pip install pyinstaller waitress
pause
exit /b 1
)
REM Check if Waitress is installed
python -c "import waitress" >nul 2>&1
py -3 -c "import waitress" >nul 2>&1
if errorlevel 1 (
echo Error: Waitress is not installed. Please run:
echo pip install waitress
echo py -3 -m pip install waitress
pause
exit /b 1
)
@ -41,8 +41,24 @@ pyinstaller --onefile ^
--hidden-import skgstat ^
--hidden-import skgstat.Variogram ^
--hidden-import skgstat.OrdinaryKriging ^
--add-data "src\gasflux\gasflux_config.yaml;src\gasflux" ^
--add-data "API_DOCUMENTATION.md;." ^
--hidden-import skgstat.DirectionalVariogram ^
--hidden-import plotly ^
--hidden-import plotly.graph_objects ^
--hidden-import plotly.express ^
--hidden-import plotly.subplots ^
--hidden-import pybaselines ^
--hidden-import scipy ^
--hidden-import scipy.odr ^
--hidden-import scipy.signal ^
--hidden-import scipy.stats ^
--hidden-import scipy.integrate ^
--hidden-import scipy.optimize ^
--hidden-import molmass ^
--hidden-import openpyxl ^
--hidden-import pyproj ^
--hidden-import jinja2 ^
--hidden-import requests ^
--hidden-import urllib3 ^
--hidden-import matplotlib ^
--hidden-import matplotlib.pyplot ^
--hidden-import matplotlib.backends ^
@ -53,7 +69,19 @@ pyinstaller --onefile ^
--hidden-import matplotlib.patches ^
--hidden-import matplotlib.text ^
--hidden-import matplotlib.transforms ^
--hidden-import tqdm ^
--hidden-import certifi ^
--hidden-import charset_normalizer ^
--hidden-import geopandas ^
--hidden-import shapely ^
--hidden-import fiona ^
--hidden-import simplekml ^
--hidden-import joblib ^
--exclude-module tkinter ^
--add-data "src\gasflux\gasflux_config.yaml;src\gasflux" ^
--add-data "src\gasflux\templates\mass_balance_template.html;src\gasflux\templates" ^
--add-data "gasflux.ini;." ^
--add-data "gasflux.ini.example;." ^
server_waitress.py
if errorlevel 1 (
@ -67,8 +95,10 @@ echo Build completed successfully!
echo Executable created: dist\GasFluxAPI.exe
echo.
echo To run the server:
echo GasFluxAPI.exe
echo GasFluxAPI.exe
echo.
echo The server will start on http://localhost:5000
echo The server will start on http://localhost:5001
echo Make sure gasflux.ini is in the same directory as the executable,
echo or configure paths via environment variables.
echo.
pause
pause

View File

@ -30,9 +30,9 @@ task_cleanup_interval = 30
# 24 hours in seconds
max_task_age = 86400
# 1 minute in seconds for successful tasks
successful_task_cleanup_age = 60
successful_task_cleanup_age = 315360000
# 1 minute in seconds for failed tasks
failed_task_cleanup_age = 60
failed_task_cleanup_age = 315360000
janitor_dry_run = false
[performance]

View File

@ -45,9 +45,17 @@ import sys
from pathlib import Path
# 设置控制台编码为 UTF-8解决 Windows 中文乱码问题)
# PyInstaller onefile 模式下 sys.stdout.buffer 可能不存在(无控制台或管道重定向),
# 此时跳过编码包装,避免 AttributeError 导致启动崩溃
import io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')
sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8')
try:
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')
except (AttributeError, ValueError):
pass
try:
sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8')
except (AttributeError, ValueError):
pass
# Add the project root and src directory to PYTHONPATH
project_root = Path(__file__).parent.absolute()

View File

@ -3,6 +3,7 @@ Download Blueprint
Handles file download endpoints.
"""
from datetime import datetime
from pathlib import Path
from flask import Blueprint, send_file, current_app
@ -10,40 +11,6 @@ from ..shared import _format_response, log_performance, logger
from ..auth import require_api_key
def _mark_task_downloaded(task_id):
"""Mark task as downloaded and schedule deletion in database."""
import sqlite3
from pathlib import Path
# Use independent database connection (not from flask.g which may be closed)
from ..db import get_db_path as get_config_db_path
db_path = get_config_db_path(current_app)
# Get cleanup age for successful tasks from config (in seconds)
successful_task_cleanup_age = current_app.config.get('SUCCESSFUL_TASK_CLEANUP_AGE', 3600)
try:
conn = sqlite3.connect(str(db_path), check_same_thread=False)
conn.execute("PRAGMA foreign_keys=ON")
conn.execute("PRAGMA busy_timeout=3000")
# Update downloaded timestamp and set deletion time based on config
conn.execute("""
UPDATE tasks
SET downloaded_at = datetime('now', '+8 hours'),
delete_after_at = datetime('now', '+8 hours', '+' || ? || ' seconds')
WHERE task_id = ?
""", (successful_task_cleanup_age, task_id))
conn.commit()
logger.info(f"Task {task_id} marked as downloaded, scheduled for deletion in {successful_task_cleanup_age} seconds")
except Exception as e:
logger.error(f"Failed to mark task {task_id} as downloaded: {str(e)}", exc_info=True)
finally:
if 'conn' in locals():
conn.close()
# Create blueprint
download_bp = Blueprint('download', __name__, url_prefix='/download')
@ -124,12 +91,19 @@ def download_file(filename):
file_size = file_path.stat().st_size
logger.info(f"Serving file: {filename} ({file_size} bytes)")
# Mark download immediately before sending file
# 记录下载时间(仅时间戳,不设置自动删除)
if task_id:
try:
_mark_task_downloaded(task_id)
from ..db import get_db
db = get_db()
db.execute(
"UPDATE tasks SET downloaded_at = datetime('now', '+8 hours') WHERE task_id = ?",
(task_id,)
)
db.commit()
logger.info(f"Task {task_id} downloaded at {datetime.now()}")
except Exception as e:
logger.error(f"Failed to mark download for task {task_id}: {str(e)}")
logger.error(f"Failed to record download for task {task_id}: {str(e)}")
response = send_file(file_path)
return response

View File

@ -99,11 +99,6 @@ def ordinary_kriging(
# np.nan_to_num(error_1s, copy=False, nan=0)
volume_error = simpsonintegrate(error_1s, x_cell_size, y_cell_size)
# Plots disabled
contour_plot = None
grid_plot = None
semivariogram_plot = None
output_text = (
f"The emissions flux of {gas.upper()} is {volume:.3f}kgh⁻¹; "
f"the cut and fill volumes of the grid are {volumepos:.3f} and {volumeneg:.3f}kgh⁻¹. "
@ -123,6 +118,20 @@ def ordinary_kriging(
"volume_error": volume_error,
}
# Generate plots (must be after krig_variables is defined)
try:
contour_plot = plotting._contour_krig_wrapper(krig_variables)
except Exception:
contour_plot = None
try:
grid_plot = plotting._heatmap_krig_wrapper(krig_variables)
except Exception:
grid_plot = None
try:
semivariogram_plot = plotting._semivariogram_plot(semivariogram, gas)
except Exception:
semivariogram_plot = None
return krig_variables, output_text, contour_plot, grid_plot, semivariogram_plot

File diff suppressed because it is too large Load Diff

View File

@ -129,7 +129,11 @@ def remove_outliers(df: pd.DataFrame, column: str, name: str):
fence_low = q1 - 3 * iqr
fence_high = q3 + 3 * iqr
fig = None # plotting disabled
try:
from . import plotting
fig = plotting._outliers_wrapper(df, column, name)
except Exception:
fig = None
outliers = df.loc[(df[column] < fence_low) | (df[column] > fence_high)]
if len(outliers) > 0:

View File

@ -5,6 +5,7 @@ from scipy import stats
import pandas as pd
import yaml
import plotly.graph_objects as go
from src.gasflux import background,plotting,processing,reporting,interpolation,pre_processing,gas
@ -123,9 +124,22 @@ class InSituSensorStrategy(SensorStrategy):
def process(self):
logger.info("Processing in-situ (point) data")
for gas in self.data_processor.gases:
self.data_processor.figs["scatter_3d"][gas] = None
self.data_processor.figs["windrose"] = None
self.data_processor.figs["wind_timeseries"] = None
# Create 3D scatter plot of flight path with gas concentration
try:
self.data_processor.figs["scatter_3d"][gas] = plotting._scatter_3d_wrapper(
self.data_processor.df, gas
)
except Exception:
self.data_processor.figs["scatter_3d"][gas] = plotting.blank_figure()
# Create wind rose and time series plots
try:
self.data_processor.figs["windrose"] = plotting._windrose_wrapper(self.data_processor.df)
except Exception:
self.data_processor.figs["windrose"] = plotting.blank_figure()
try:
self.data_processor.figs["wind_timeseries"] = plotting._time_series_wrapper(self.data_processor.df)
except Exception:
self.data_processor.figs["wind_timeseries"] = plotting.blank_figure()
class SpatialProcessingStrategy(ABC):

View File

@ -27,13 +27,22 @@ def mass_balance_report(
"""Generate a mass balance report (plots disabled)."""
template_path = Path(__file__).parent / "templates" / "mass_balance_template.html"
# Plots disabled -> use empty strings
# Convert figures to HTML strings
def _fig_to_html(fig) -> str:
"""Safely convert a plotly figure to HTML string."""
if fig is None:
return ""
try:
return fig.to_html(full_html=False, include_plotlyjs=True)
except Exception:
return ""
plot_htmls = {
"3D": "",
"krig": "",
"windrose": "",
"wind": "",
"background": "",
"3D": _fig_to_html(threed_fig),
"krig": _fig_to_html(krig_fig),
"windrose": _fig_to_html(windrose_fig),
"wind": _fig_to_html(wind_fig),
"background": _fig_to_html(background_fig),
}
summary_data = {