hiveSql 遷移spark2.4時報錯Error in query: Window function row_number() requires window to be ordered, please add ORDER BY clause

hivesql對語法檢查較弱

像下面的語法 hive是可以通過的 partition by 后沒有跟order by 

row_number() over(partition by buvid,version_code,app_id) as rn

原因看下hive 源碼(hive 已經做了補充)

 

 spark中 看下代碼

  /**
   * Check and add order to [[AggregateWindowFunction]]s.
   */
  object ResolveWindowOrder extends Rule[LogicalPlan] {
    def apply(plan: LogicalPlan): LogicalPlan = plan resolveExpressions {
      case WindowExpression(wf: WindowFunction, spec) if spec.orderSpec.isEmpty =>
        failAnalysis(s"Window function $wf requires window to be ordered, please add ORDER BY " +
          s"clause. For example SELECT $wf(value_expr) OVER (PARTITION BY window_partition " +
          s"ORDER BY window_ordering) from table")
      case WindowExpression(rank: RankLike, spec) if spec.resolved =>
        val order = spec.orderSpec.map(_.child)
        WindowExpression(rank.withOrder(order), spec)
    }
  }

這里我們注釋掉就行 如果對排序不是很看重 對結果集順序無所謂

 /**
   * Check and add order to [[AggregateWindowFunction]]s.
   */
  object ResolveWindowOrder extends Rule[LogicalPlan] {
    def apply(plan: LogicalPlan): LogicalPlan = plan resolveExpressions {
//      case WindowExpression(wf: WindowFunction, spec) if spec.orderSpec.isEmpty =>
//        failAnalysis(s"Window function $wf requires window to be ordered, please add ORDER BY " +
//          s"clause. For example SELECT $wf(value_expr) OVER (PARTITION BY window_partition " +
//          s"ORDER BY window_ordering) from table")
      case WindowExpression(rank: RankLike, spec) if spec.resolved =>
        val order = spec.orderSpec.map(_.child)
        WindowExpression(rank.withOrder(order), spec)
    }
  }