Learning from video, vehicle sensors, and screens, from early video retrieval and vehicle behavior mining to agents that operate computers.
Today this thread runs through computer use, where agents operate full desktop environments: ChainWorld (Siu et al., 2026) composes atomic OSWorld tasks into long-horizon desktop workloads, and Spine-Branch (Zhang et al., 2026) coordinates teams of computer-use agents. See also Benchmarks and Evals and Agent Teams.
Back in college, I worked on our team’s entry to TRECVID 2007(Yuan et al., 2007), an annual benchmark for content-based video retrieval run by the National Institute of Standards and Technology (NIST). The system covered three tasks. For shot boundary detection, it combined block-based color histograms with a motion detector that suppressed false alarms from camera and object motion, followed by a SIFT-based post-processing step. For high-level feature extraction (i.e., concept detection), it learned concept distributions over image regions with multi-label, multi-feature learning, added shape and spatial features, and fused scores both within and across concepts. For video search, it supported both automatic and interactive retrieval.
Back in college, in 2008, I worked on self-driving vehicles, mining vehicle behaviors from raw multi-sensor data (Liu et al., 2008). The resulting qualitative descriptions of vehicle state feed downstream tasks like scenario parsing, vehicle surveillance, and forward-collision warning. Earlier approaches either assumed the number of behavior categories was known in advance or had to rediscover it by hand whenever the training data changed. We instead treat behaviors as the hidden states of a hidden Markov model and choose the number of states by cross-validated likelihood with a complexity penalty, so the behavior categories and their number are learned together. Experiments on real sensor data bear out the approach.
@article{zhang2026spinebranch,title={Spine-Branch Coordination for Multi-agent Computer Use},author={Zhang, Mian and Sharma, Manasi and Zhang, Sheng and Yang, Minglai and Shi, Kejian and Liu, Ying and Chen, Zhiyu Zoey and Zhang, Daniel Yue},journal={arXiv preprint arXiv:2608.22077},year={2026},month=aug,}
@article{siu2026chainworld,title={{ChainWorld}: Composing Long-Horizon Desktop Workloads from Atomic {OSWorld} Tasks},author={Siu, Vincent and Sharma, Manasi and Song, Dawn and Zhang, Daniel Yue and Wang, Chenguang and Liu, Ying},journal={arXiv preprint arXiv:2606.21654},year={2026},month=jun,}
@inproceedings{liu2008vehicle,title={Automatic Mining of Vehicle Behaviors with an Unknown Number of Categories},author={Liu, Ying and Zhang, Hao and Meng, Huadong and Wang, Xiqin},booktitle={IEEE International Conference on Intelligent Transportation Systems (ITSC)},address={Beijing, China},pages={247--252},year={2008},month=oct,doi={10.1109/ITSC.2008.4732543},}
@inproceedings{yuan2007thuicrc,title={{THU} and {ICRC} at {TRECVID} 2007},author={Yuan, Jinhui and Guo, Zhishan and Lv, Li and Wan, Wei and Zhang, Teng and Wang, Dong and Liu, Xiaobing and Liu, Cailiang and Zhu, Shengqi and Wang, Duanpeng and Pang, Yang and Ding, Nan and Liu, Ying and Wang, Jiangping and Zhang, Xiujun and Tie, Xiaozheng and Wang, Zhikun and Wang, Huiyi and Xiao, Tongchun and Liang, Yinyu and Li, Jianmin and Lin, Fuzong and Zhang, Bo and Li, JianGuo and Wu, WeiXin and Tong, XiaoFeng and Ding, DaYong and Chen, YuRong and Wang, Tao and Zhang, Yimin},booktitle={TREC Video Retrieval Evaluation (TRECVID) Workshop},address={Gaithersburg, MD},year={2007},month=nov,}