Cornell researchers have released nine hours of labeled underwater video to help computer scientists build better tools for tracking fish behavior outside controlled laboratory settings, according to a university account of the project.
The dataset, called WildFin, grew out of doctoral candidate Abigail Grassick's work studying fish communities near coral formations off Curaçao. Existing computer vision systems could not reliably follow individual fish in her field footage, much less identify behavior such as feeding or evading predators.
Field footage poses a harder test
Grassick, a computational biology researcher, and colleagues labeled the videos frame by frame, marking behaviors including foraging, feeding and fighting. The material includes footage of groups of fish around individual coral formations, as well as videos in which a diver followed a single fish. Colleagues at the University of Colorado Boulder collected and annotated the diver-followed footage; all of the videos came from sites off Curaçao.
The field conditions differ from those in a laboratory. Fish move through groups, lighting changes and camera angles shift. Jennifer Sun, a Cornell computer science professor who has used vision systems to interpret animal behavior in labs, said it can be difficult even to determine whether a fish in one frame is the same animal in the next.
The team spent about 1,400 hours collecting footage in the field and 600 hours annotating it to produce the nine-hour dataset, Grassick said. Researchers then used the footage to fine-tune several standard computer vision models to classify behavior. Performance remained poor, so the dataset is a starting point for improvement, not a system already capable of reliably identifying fish behavior in the wild.
Grassick presented the team's paper, “WildFin: An In-the-Wild Dataset for Fish Behavioral Recognition,” at the European Conference on Computer Vision on Sept. 8 in Malmö, Sweden. The project site makes the research materials available to other researchers.
Why the researchers want more video
Andrew Hein, a Cornell computational biology professor and co-author, said current models are trained largely on images and videos that do not resemble scientific field footage. Better tools could help ecologists analyze the expensive and time-consuming footage they already collect, and potentially make citizen-science videos useful for monitoring ecosystems and recording rare species.
The team is asking other ecologists to share field videos to expand the range of conditions represented in future training data. Sun said those archives often sit unused after an initial project, though preparing varied footage for machine learning remains a challenge.
The project involved Cornell students and collaborators at Colorado Boulder and the Howard Hughes Medical Institute. The National Science Foundation provided partial research support, and NVIDIA's Academic Grant Program supported hardware. The Cornell release did not report a validated model accuracy level or demonstrate that the approach can yet be applied across other environments or species.
FingerLakes1.com is the region’s leading all-digital news publication. The company was founded in 1998 and has been keeping residents informed for more than two decades. Have a lead? Send it to news@fingerlakes1.com.