I have fixed this problem by replacing ReorderableListView by the package flutter_reorderable_list, it accepts a child (Widget) unlike ReorderableListView which accepts children (List)
Its implementation is a bit more complex. You can look at the example to get some understanding of implementing it correctly.